Performing predictive inferences using multiple predictive models

ABSTRACT

Method, apparatus and computer program product for performing a cross-model predictive inference to generate a cross-model predictive output for a plurality of predictive inputs using a plurality of predictive models. For example, the apparatus includes at least one processor and at least one non-transitory memory including program code. The at least one non-transitory memory and the program code are configured to, with the at least one processor, obtain a model selection probability distribution which defines, for each predictive model, a respective selection probability score; obtain, for each predictive model, respective cross-model normalization data; for each predictive input, determine a cross-model predictive score; and determine, based on each determined cross-model predictive score, the cross-model predictive output.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S. PatentApplication No. 16/869,439, titled “PERFORMING PREDICTIVE INFERENCESUSING MULTIPLE PREDICTIVE MODELS,” and filed May 7, 2020, which claimspriority to U.S. Provisional Application Serial No. 62/844,522, titled“PERFORMING PREDICTIVE INFERENCES USING MULTIPLE PREDICTIVE MODELS,”filed May 7, 2019, the contents of which are incorporated herein byreference in their entirety.

BACKGROUND

Applicant has identified many deficiencies and problems associated withexisting methods, apparatus, and systems related to performingpredictive inferences using multiple (i.e., two or more) predictivemodels. Through applied effort, ingenuity, and innovation, many of theseidentified deficiencies and problems have been solved by developingsolutions that are in accordance with embodiments of the presentdisclosure, many examples of which are described in detail herein.

BRIEF SUMMARY

In general, embodiments of the present disclosure provide methods,apparatus, systems, computing devices, and/or the like for performingpredictive inferences using multiple (i.e., two or more) predictivemodels by using one or more of cross-model predictive inference,cross-model predictive score generation, cross-model score verification,and cross-model normalization.

In accordance with one aspect, an apparatus is provided. The apparatuscomprises at least one processor and at least one non-transitory memorycomprising program code. The at least one non-transitory memory and theprogram code are configured to obtain a model selection probabilitydistribution, wherein the model selection probability distributiondefines, for each predictive model of a plurality of predictive models,a respective selection probability score; obtain, for each predictivemodel of the plurality of predictive models, respective cross-modelnormalization data; determine a cross-model predictive score for eachpredictive input of the plurality of predictive inputs, whereindetermining the cross-model predictive score for the predictive inputcomprises: determine, based on a weighted random selection for thepredictive input, a respective selected predictive model of theplurality of predictive models, wherein at least one weighted randomselection parameter is determined based on the model selectionprobability distribution; determine one or more model-specificpredictive scores for the predictive input by applying the respectiveselected predictive model to the predictive input; and determine thecross-model predictive score for the predictive input by transformingthe one or more model-specific predictive scores associated with thepredictive input using the cross-model normalization data for theselected predictive model associated with the predictive input;determine, based on each cross-model predictive score for a predictiveinput of the plurality of predictive inputs, the cross-model predictiveoutput for the plurality of predictive models; generate, based on thecross-model predictive output, an electronic communication for apromotional outreach related to a merchant of goods or services; andtransmit the electronic communication to a computing device tofacilitate rendering of data associated with the electroniccommunication via a graphical interface of the computing device.

In some embodiments, the at least one non-transitory memory and theprogram code are configured to identify, from the plurality ofpredictive models, a champion predictive model and one or morechallenger predictive models. In some embodiments, the at least onenon-transitory memory and the program code are configured to determine,for each predictive input of a plurality of predictive inputs, aninitial ranking predictive score by applying the champion predictivemodel to the predictive input, wherein determining each cross-modelpredictive score for a predictive input of the plurality of predictiveinputs comprises determining each cross-model predictive score in apredictive score generation order; and wherein the predictive scoregeneration order is determined based on each initial ranking predictivescore associated with a predictive input of the plurality of predictiveinputs. In some embodiments, the champion predictive model has arespective selection probability score that is higher than therespective selection probability score for each challenger predictivemodel of the one or more challenger predictive models.

In some embodiments, the at least one non-transitory memory and theprogram code are configured to simultaneously generate a score for thechampion predictive model and respective scores for the one or morechallenger predictive models. Additionally, in some embodiments, the atleast one non-transitory memory and the program code are configured tocompare the respective scores for the one or more challenger predictivemodels to the score for the champion predictive model. In someembodiments, the at least one non-transitory memory and the program codeare configured to evaluate the score for the champion predictive modeland the respective scores for the one or more challenger predictivemodels based on two or more objective functions. In some embodiments,the at least one non-transitory memory and the program code areconfigured to select the champion predictive model or the one or morechallenger predictive models based on an evaluation of the score for thechampion predictive model and the respective scores for the one or morechallenger predictive models.

In some embodiments, the at least one non-transitory memory and theprogram code are configured to identify the champion predictive model,and wherein identifying the champion predictive model comprises:identifying, for each predictive model of the plurality of predictivemodels, a retrospective predictive success score, wherein theretrospective predictive success score for each predictive model definesa retrospective prediction-outcome correspondence for the predictivemodel between one or more retrospective predictive scores generated bythe predictive model and one or more retrospective ground-truthpredictive outcomes corresponding to the one or more past predictivescores; and determining the champion predictive model based on eachretrospective predictive success score associated with a predictivemodel of the plurality of predictive models. In some embodiments, the atleast one non-transitory memory and the program code are configured to,for each predictive input of the plurality of predictive inputs:identify one or more unselected predictive models for the predictiveinput, wherein the one or more unselected predictive models for thepredictive input comprise each predictive model in the plurality ofpredictive models other than the selected predictive model for thepredictive input; determine, for each unselected predictive model of theone or more unselected predictive models, a verification predictivescore by applying the unselected predictive model to the predictiveinput; obtain, subsequent to expiration of a threshold prospectiveperformance time interval after determining the cross-model predictiveoutput, a prospective ground-truth success outcome for the predictiveinput corresponding to the threshold prospective performance timeinterval; and determine a prospective predictive success score for eachpredictive model of the plurality of predictive models, wherein theprospective predictive success score for the selected predictive modelis determined based on a recent ground-truth success outcome for thepredictive input and the cross-model predictive score for the predictiveinput, and wherein each prospective predictive success score for anunselected predictive model of the one or more unselected predictivemodel is determined based on the recent ground-truth success outcome forthe predictive input and the verification predictive score associatedwith the unselected predictive model.

In some embodiments, the at least one non-transitory memory and theprogram code are configured to adjust the selection probability scorefor each predictive model of the plurality of predictive models based onthe prospective predictive success score for the predictive model. Insome embodiments, each cross-model normalization data for a predictivemodel of the plurality of predictive models defines one or morecross-model conversion operations for the predictive model; the one ormore cross-model conversion operations for a respective predictive modelare configured to convert the cross-model predictive output for therespective predictive model each having an output-specific range to across-model value having a cross-model range; and each cross-modelpredictive score for a predictive model of the plurality of predictivemodels is determined based on a respective cross-model value for thepredictive model. In some embodiments, determining the cross-modelpredictive output for the plurality of predictive models comprises:determining a predictive ranking of the plurality of predictive inputsbased on each cross-model predictive score associated with a predictiveinput of the plurality of predictive inputs; determining the cross-modelpredictive output based on the predictive ranking of the plurality ofpredictive inputs; generating, based on the cross-model predictiveoutput, an electronic communication for a promotional outreach relatedto a merchant of goods or services; and transmitting the electroniccommunication to a computing device to facilitate rendering of dataassociated with the electronic communication via a graphical interfaceof the computing device.

In some embodiments, each predictive input of the plurality ofpredictive inputs is associated with a candidate merchant identifier ofa plurality of candidate merchant identifiers; each candidate merchantidentifier of the plurality of candidate merchant identifiers isassociated with a candidate merchant data structure of a plurality ofcandidate merchant data structures; each predictive input associatedwith a candidate merchant identifier comprises the candidate merchantdata structure associated with the candidate merchant identifier; eachcross-model predictive score for a predictive input of the plurality ofpredictive inputs indicates a merchant promotional outreach interestprediction for the candidate merchant identifier associated with thepredictive input; and the cross-model predictive output indicates athreshold number of merchant identifiers of the plurality of candidatemerchant identifiers whose corresponding merchant promotional outreachinterest prediction exceeds is highest among each merchant promotionaloutreach interest prediction associated with a candidate merchantidentifier of the plurality of candidate merchant identifiers. In someembodiments, determining the cross-model predictive output for theplurality of predictive models comprises: determining a predictiveranking of the plurality of predictive inputs based on each cross-modelpredictive score associated with a predictive input of the plurality ofpredictive inputs; determining the cross-model predictive output basedon the predictive ranking of the plurality of predictive inputs;generating, based on the cross-model predictive output, an electroniccommunication for a promotional outreach related to a merchant of goodsor services; and transmitting the electronic communication to acomputing device to facilitate rendering of data associated with theelectronic communication via a graphical interface of the computingdevice.

In some embodiments, each predictive input of the plurality ofpredictive inputs is associated with a candidate merchant identifier ofa plurality of candidate merchant identifiers; each candidate merchantidentifier of the plurality of candidate merchant identifiers isassociated with a candidate merchant data structure of a plurality ofcandidate merchant data structures; each predictive input associatedwith a candidate merchant identifier comprises the candidate merchantdata structure associated with the candidate merchant identifier; eachcross-model predictive score for a predictive input of the plurality ofpredictive inputs indicates a merchant promotional outreach interestprediction for the candidate merchant identifier associated with thepredictive input; and the cross-model predictive output indicates athreshold number of merchant identifiers of the plurality of candidatemerchant identifiers whose corresponding merchant promotional outreachinterest prediction exceeds is highest among each merchant promotionaloutreach interest prediction associated with a candidate merchantidentifier of the plurality of candidate merchant identifiers. In someembodiments, determining the selected predictive model for a respectivepredictive input comprises: determining, based on the weighted randomselection, a first randomly-selected predictive model for the predictiveinput.

In some embodiments, the at least one non-transitory memory and theprogram code are configured to identify the cross-model predictive scorefor the predictive input generated by the first randomly-selectedpredictive model; determine a first candidate cross-model predictivescore by applying the first randomly-selected predictive model to thepredictive input to generate one or more first candidate model-specificpredictive scores and transforming the one or more first candidatemodel-specific predictive scores to the first candidate cross-modelpredictive score based on the cross-model normalization data for thefirst randomly-selected predictive model; determine whether the firstcandidate cross-model predictive score for the predictive inputsatisfies cross-model score adoption data for the plurality ofpredictive models; and in response to determining that the firstcandidate cross-model predictive score for the predictive inputsatisfies the cross-model score adoption data for the plurality ofpredictive models, adopt the first candidate cross-model predictivescore as the cross-model predictive score for the predictive input.

In some embodiments, the at least one non-transitory memory and theprogram code are configured to, in response to determining that thefirst candidate cross-model predictive score for the predictive inputfails to satisfy the cross-model score adoption data for the pluralityof predictive models: identify a champion predictive model of theplurality of predictive models; determine a second candidate cross-modelpredictive score by applying the champion predictive model to thepredictive input to generate one or more second candidate model-specificpredictive scores and transforming the one or more second candidatemodel-specific predictive scores to the second candidate cross-modelpredictive score based on the cross-model normalization data for thechampion predictive model; determine whether the second candidatecross-model predictive score for the predictive input satisfiescross-model score adoption data for the plurality of predictive models;and in response to determining that the second candidate cross-modelpredictive score for the predictive input satisfies the cross-modelscore adoption data for the plurality of predictive models, adopt thesecond candidate cross-model predictive score as the cross-modelpredictive score for the predictive input.

In some embodiments, the at least one non-transitory memory and theprogram code are configured to, in response to determining that thefirst candidate cross-model predictive score fails to satisfy thecross-model score adoption data for the plurality of predictive models:perform a second weighted random selection for the predictive input todetermine a second randomly-selected predictive model of the pluralityof predictive models; determine a second candidate cross-modelpredictive score by applying the second randomly-selected predictivemodel to the predictive input to generate one or more second candidatemodel-specific predictive scores and transforming the one or more secondcandidate model-specific predictive scores to the second candidatecross-model predictive score based on the cross-model normalization datafor the second randomly-selected predictive model; determine whether thesecond candidate cross-model predictive score for the predictive inputsatisfies cross-model score adoption data for the plurality ofpredictive models; and in response to determining that the secondcandidate cross-model predictive score for the predictive inputsatisfies the cross-model score adoption data for the plurality ofpredictive models, adopt the second candidate cross-model predictivescore as the cross-model predictive score for the predictive input.

In some embodiments, the at least one non-transitory memory and theprogram code are configured to, in response to determining that thefirst candidate cross-model predictive score for the predictive inputfails to satisfy the cross-model score adoption data for the pluralityof predictive models, modifying the plurality of predictive inputs toeliminate the predictive input. In some embodiments, the weighted randomselection for a predictive input of the plurality of predictive inputsis characterized by one or more weighted random selection parameters forthe predictive input.

The above summary is provided merely for purposes of summarizing someexample embodiments to provide a basic understanding of some aspects ofthe disclosure. Accordingly, it will be appreciated that theabove-described embodiments are merely examples and should not beconstrued to narrow the scope or spirit of the disclosure. It will beappreciated that the scope of the disclosure encompasses many potentialembodiments in addition to those here summarized, some of which will befurther described below.

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described some embodiments in general terms, references willnow be made to the accompanying drawings, which are not necessarilydrawn to scale, and wherein:

FIG. 1 is an example system architecture within which embodiments of thepresent disclosure may operate.

FIG. 2 is a schematic diagram of an example apparatus for a predictiveanalysis server in accordance with one embodiment of the presentdisclosure.

FIG. 3 is a schematic diagram of an example apparatus for a clientcomputing device in accordance with one embodiment of the presentdisclosure.

FIG. 4 is a flow diagram of a process for performing a cross-modelpredictive inference in accordance with one embodiment of the presentdisclosure.

FIG. 5 is a flow diagram of a process for performing cross-modelpredictive score generation for a predictive input in accordance withone embodiment of the present disclosure.

FIG. 6 is a flow diagram of a process for selecting a predictive modelfor a predictive input in accordance with one embodiment of the presentdisclosure.

FIG. 7 is a flow diagram of a process for performing cross-modelpredictive verification for a predictive input in accordance with oneembodiment of the present disclosure.

FIG. 8 is an exemplary electronic interface in accordance with oneembodiment of the present disclosure.

FIG. 9 is another exemplary electronic interface in accordance with oneembodiment of the present disclosure.

FIG. 10 is yet another exemplary electronic interface in accordance withone embodiment of the present disclosure.

FIG. 11 is yet another exemplary electronic interface in accordance withone embodiment of the present disclosure.

FIG. 12 is yet another exemplary electronic interface in accordance withone embodiment of the present disclosure.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

Various embodiments of the present disclosure are described more fullyhereinafter with reference to the accompanying drawings, in which some,but not all embodiments of the disclosure are shown. Indeed, thedisclosure may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein. Rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements. The term “or” is used herein in both the alternativeand conjunctive sense, unless otherwise indicated. The terms“illustrative,” “example,” and “exemplary” are used to be examples withno indication of quality level. Like numbers refer to like elementsthroughout.

The term “comprising” means “including but not limited to,” and shouldbe interpreted in the manner it is typically used in the patent context.Use of broader terms such as comprises, includes, and having should beunderstood to provide support for narrower terms such as consisting of,consisting essentially of, and comprised substantially of.

The phrases “in one embodiment,” “according to one embodiment,” and thelike generally mean that the particular feature, structure, orcharacteristic following the phrase may be included in at least oneembodiment of the present disclosure, and may be included in more thanone embodiment of the present disclosure (importantly, such phrases donot necessarily refer to the same embodiment).

Overview

Various embodiments of the present disclosure address technicalchallenges related to efficiency and reliability of predictive dataanalysis systems by enabling various techniques related to cross-modelpredictive inference. The field of predictive data analysis has hadenormous recent successes which has led to the development of variouspredictive models (e.g., neural network models, Bayesian inferencemodels, etc.). Each predictive model is typically configured to processpredictive inputs of a particular format in accordance with a set ofpredictive parameters to generate a predictive output. Depending ontheir respective architectural complexities, operational requirements,and training complexities, different predictive models are utilized toprocess different predictive inferences. Thus, while a first predictivemodel may be suited to perform a first predictive inference having afirst set of properties, the same first predictive model may beill-suited to perform a second predictive inference having a second setof properties. On the other hand, a second predictive model may bebetter suited to perform the second predictive inference. For example, apredictive model that requires a high amount of training may beill-suited for a predictive task that has limited training data and/orthat should be performed with minimal operational load (e.g., apredictive task configured to be performed on a device with minimalprocessing and/or storage capacities). As another example, a predictivemodel that utilizes numeric transformations may be ill-suited for apredictive task that deals with a logically coherent real-world domain(e.g., the medical domain).

Using an ill-suited predictive model to perform a predictive task maylead to substantial efficiency and reliability drawbacks for predictivedata analysis systems. For example, using predictive models that haveexcessive operational requirements may lead to significant efficiencydrawbacks for predictive data analysis systems. As another example,using a predictive model that requires a high amount of training toperform a predictive task that should be performed with minimaloperational load will both lead to efficiency drawbacks and make theoverall for predictive data analysis systems unreliable due to highsystem failures and/or high response times. As a further example, usingpredictive models that require substantial training for predictive tasksnot conducive to substantial training may undermine accuracy of suchpredictive models, and thus undermine the reliability of predictive dataanalysis systems utilizing such models.

Various embodiments of the present disclosure address the above-notedefficiency and reliability challenges for predictive data analysissystems stemming from predictive model diversity by introducing varioustechniques related to cross-model predictive inference that enabledetecting optimal predictive models for individual predictive inputs andapplying the detected optimal predictive models to the noted predictiveinputs. Examples of the noted techniques include cross-model predictiveinference, cross-model predictive score generation, cross-model scoreverification, and cross-model normalization. For example, by utilizingsome aspects of cross-model predictive inference and some aspects ofcross-model predictive score generation, various embodiments of thepresent disclosure enable selectively utilizing multiple predictivemodels to process multiple predictive inputs, thus enabling selection ofthe most optimal predictive model for each predictive input whenperforming cross-model predictive inferences. As another example, byutilizing some aspects of cross-model score verification, variousembodiments of the present disclosure enable determining selectionprobabilities of individual predictive models based on empiricalobservations about correspondence of predictions generated by thosemodels and real-world predictions, thus enabling improved detection ofoptimal predictive models for individual predictive inputs whenperforming cross-model predictive inferences. As yet another example, byutilizing some aspects of cross-model normalization, various embodimentsof the present disclosure enable generating normalized outputs having across-model format based on model-specific outputs each having amodel-specific format, thus enabling better integration andinteroperability between various predictive models when performingcross-model predictive inferences.

Various embodiments of the present disclosure relate to cross-modelverification and/or cross-model normalization of various predictivemodels in multi-model predictive systems. For example, given threemodels each predicting a ranking of the same set of data, it may beimpractical and/or impossible to test all the three predictive models,at least in part because of the overlap in how those models perform theranking prediction. As another example, given 100 predictive inputscorresponding to 100 marketing leads for merchants of goods or services,and further given three predictive models, it may be impractical and/orimpossible to provide the marketing leads to three different marketingagents determined by the three predictive models. This may be because,as soon a first predicted team acts with respect to a particular inputlead, that lead may be unavailable for further action by other predictedteams. As a result, there is a technical need for cross-modelverification and normalization across various predictive models. Variousembodiments of the present disclosure address this technical need byenabling techniques for one or more of cross-model predictive inference,cross-model predictive score generation, cross-model score verification,and cross-model normalization. In doing so, the noted embodimentsimprove accuracy and reliability of predictive models in multi-modelpredictive system, as well as efficiency of model verification andnormalization in such systems.

Additionally, various embodiments of the present disclosure provide apredictive analysis system for selectively utilizing multiple predictivemodels to process multiple predictive inputs related to electroniccommunications regarding outreach to merchants of goods or services. Forexample, in order to effectively determine whether to generate andtransmit an electronic communication regarding outreach for a particularmarket of goods or services and/or particular merchants of goods orservices, efficient testing and/or validation of predictive models formarketing leads related to a given market and/or merchants of goods orservices are provided. The testing and/or validation of the predictivemodels can include cross-model verification and/or cross-modelnormalization of the predictive models.

In certain embodiments, a predictive model can employ a mapping ofmerchant identifiers with respect to different markets for goods orservices to facilitate a ranking of candidate merchants for particulargoods or services. In an embodiment, a predictive model is a retrainedversion and/or a modified version of one or more previous predictivemodels. In another embodiment, one or more predictive models solve for adifferent use case for goods or services as compared to one or moreother predictive models.

In certain embodiments, an optimal ranking model of candidate merchantsfor a particular good or service can be determined. In anotherembodiment, a predictive model with a highest score can be employed todetermine one or more merchants for a merchant promotional outreach. Incertain embodiments, an electronic communication for a merchantpromotional outreach can be generated and/or transmitted to a consumerdevice to facilitate rendering of data for the merchant promotionaloutreach via an electronic interface of the consumer device.

Accordingly, a number of computing resources employed by a predictiveanalysis system for transmitting electronic communications to one ormore consumer devices can be reduced. Additionally or alternatively, fora predictive analysis system that provides predictive analysis viapredictive models and/or manages a predictive model repository tofacilitate predictive analysis via predictive models, a number of memoryresources related to candidate merchants and/or goods or services for amerchant promotional outreach can be reduced.

Various embodiments additionally or alternatively improve performance ofa processor configured to execute one or more operations associated withcross-model predictive inference, cross-model predictive scoregeneration, cross-model score verification, and/or cross-modelnormalization. For example, various embodiments improve processing speedand/or reduce a number of computational resources associated withcross-model predictive inference, cross-model predictive scoregeneration, cross-model score verification, and/or cross-modelnormalization. Furthermore, various embodiments additionally oralternatively improve training of a machine learning network (e.g., aneural network) associated with cross-model predictive inference,cross-model predictive score generation, cross-model score verification,and/or cross-model normalization. For example, various embodimentsadditionally or alternatively provide improved or reduced parameters,improved values, improved weights, and/or improved thresholds for amachine learning network (e.g., a neural network) associated withcross-model predictive inference, cross-model predictive scoregeneration, cross-model score verification, and/or cross-modelnormalization.

Various embodiments of the present disclosure relate to gradual testingand/or roll-out of new predictive models. For example, given model A(e.g., an incumbent predictive model) and model Y (e.g., a challengerpredictive model), various embodiments of the present disclosuredisclose gradually testing Y over time and, given the test results,allocating a proportional portion of predictive tasks to model Y. Inenabling gradual testing and/or roll-out of new predictive models,various embodiments of the present disclosure increase efficiency and/oraccuracy associated with model roll-out, while enhancing the overallefficiency and reliability of a multi-model predictive system.

Various embodiments additionally or alternatively improve performance ofa processor configured to execute one or more operations associated withgradual testing and/or roll-out of new predictive models. For example,various embodiments improve processing speed and/or reduce a number ofcomputational resources associated with gradual testing and/or roll-outof new predictive models. Furthermore, various embodiments additionallyor alternatively improve training of a machine learning network (e.g., aneural network) associated with gradual testing and/or roll-out of newpredictive models. For example, various embodiments additionally oralternatively provide improved or reduced required parameters, improvedvalues, improved weights, and/or improved thresholds for a machinelearning network (e.g., a neural network) associated with gradualtesting and/or roll-out of new predictive models. As such, variousembodiments disclosed herein provide a technical improvement associatedwith machine learning.

Various embodiments additionally or alternatively improve efficiency ofprocessing of predictive models. For instance, various embodimentsadditionally or alternatively provide for testing and/or evaluating twoor more predictive models in parallel to provide more efficientprocessing and/or a reduced number of computational resources fortesting predictive models. In an example embodiment, three predictivemodels can be tested in parallel and the three predictive models can beevaluated based on two or more objective functions. For example, thethree predictive models can be evaluated based on a first objectivefunction (e.g., a purchase frequency for food and drink), a secondobjective function (e.g., incremental revenue at a particulargeographical location), and a third objective function (e.g., revenuefor a geographical division).

Definitions

As used herein, the terms “data,” “content,” “digital content,” “digitalcontent object,” “information,” and similar terms may be usedinterchangeably to refer to data capable of being transmitted, received,and/or stored in accordance with embodiments of the present disclosure.Thus, use of any such terms should not be taken to limit the spirit andscope of embodiments of the present disclosure. Further, where acomputing device is described herein to receive data from anothercomputing device, it will be appreciated that the data may be receiveddirectly from another computing device or may be received indirectly viaone or more intermediary computing devices, such as, for example, one ormore servers, relays, routers, network access points, base stations,hosts, and/or the like (sometimes referred to herein as a “network”).Similarly, where a computing device is described herein to send data toanother computing device, it will be appreciated that the data may besent directly to another computing device or may be sent indirectly viaone or more intermediary computing devices, such as, for example, one ormore servers, relays, routers, network access points, base stations,hosts, and/or the like.

The term “circuitry” should be understood broadly to include hardwareand, in some embodiments, software for configuring the hardware. Withrespect to components of the apparatus, the term “circuitry” as usedherein should therefore be understood to include particular hardwareconfigured to perform the functions associated with the particularcircuitry as described herein. For example, in some embodiments,“circuitry” may include processing circuitry, storage media, networkinterfaces, input/output devices, and the like.

The term “obtain” may refer to electronic retrieval (e.g., from local orremote memory or other storage such as a repository, and the like). Theterm “obtain” may refer to electronic receipt (e.g., receiving via alocal communication bus, receiving from a remote computing device orrepository via a communication network, and the like).

The term “user” should be understood to refer to an individual, group ofindividuals, business, organization, and the like. The users referred toherein may access a predictive analysis server using client computingdevices (as defined herein).

The term “client computing device” refers to computer hardware and/orsoftware that is configured to access a service made available by aserver. The server is often (but not always) on another computer system,in which case the client computing device accesses the service by way ofa network. Client computing devices may include, without limitation,smart phones, tablet computers, laptop computers, wearables, personalcomputers, enterprise computers, and the like.

The term “cross-model predictive inference” refers to acomputer-implemented process for determining a prediction based on apredictive input using a predictive framework that utilizes two or morepredictive models. For example, a cross-model predictive inference couldinclude selecting a particular predictive model of two or morepredictive models to apply to a particular predictive input and applyingthe particular predictive model to the particular predictive input todetermine a particular prediction for the particular predictive input.As another example, a cross-model predictive inference could includeapplying n predictive models of multiple predictive models to aparticular predictive input in order to determine a particularpredictive output based on the particular predictive input, where n =>2.

The term “cross-model predictive output” refers to a collection of oneor more data items that indicate a prediction determined using across-model predictive inference based on a predictive input. Forexample, a cross-model predictive inference could include applying aselected predictive model of two or more predictive models to apredictive input in order to generate a corresponding cross-modelpredictive output. As another example, a cross-model predictiveinference could include applying n predictive models of multiplepredictive models on a particular predictive input in order to determinea cross-model predictive output for the predictive input, where n => 2.The term “predictive input” refers to a collection of one or more dataitems that indicate an input to a predictive process, e.g., an input toa cross-model predictive inference. For example, a particular predictiveinput may include information about a particular merchant identifier,where the particular predictive input is configured to be supplied as aninput to a particular predictive process, and wherein the particularpredictive process is configured to generate a merchant promotionaloutreach interest prediction for the particular merchant identifier. Asanother example, a particular predictive input may include informationabout a particular patient, where the particular predictive input isconfigured to be supplied as an input to a particular predictiveprocess, and wherein the particular predictive process is configured togenerate a predictive health score for the particular patient. As yetanother example, a particular predictive input may include informationabout a particular consumer, where the particular predictive input isconfigured to be supplied as an input to a particular predictiveprocess, and wherein the particular predictive process is configured togenerate a promotion interest score for the particular consumer.

The term “predictive model” refers to a collection of one or more dataitems that indicate one or more predictive operations configured to beperformed on a predictive input in order to determine a model-specificpredictive score. Examples of predictive models include regressionmodels, Bayesian inference models, neural network models, machinelearning models, etc. Examples of data items characterizing a particularpredictive model include data items associated with parameters of thepredictive model, data items indicating a meta-structure (e.g., numberof layers) of the particular predictive model, data indicatingparticular operations (e.g., particular non-linear operations)associated with the particular predictive model, etc.

The term “selection of a predictive model” for a predictive input refersto programmatic selection of a predictive model as the predictive modelused for the predictive input. For example, selection of a predictivemodel for a predictive input may include determining a model-specificpredictive score for the predictive input based on the predictive model,where the model-specific predictive score is used to determine across-model predictive score for the predictive input.

The term “model selection probability distribution” refers to acollection of one or more data items that indicate, for each predictivemodel among two or more predictive models, a selection probabilityscore. For example, a particular model selection probabilitydistribution for a group of three predictive models may indicate thefollowing selection probability scores for the three predictive models:50% selection probability score for a first predictive model of thethree predictive models, 40% selection probability score for a secondpredictive model of the three predictive models, and 10% selectionprobability score for a third predictive model of the three predictivemodels.

The term “selection probability score” refers to a collection of one ormore data items that indicate, for a corresponding predictive model oftwo or more predictive models, a preferred ratio for selection of thecorresponding predictive model when the two or more predictive modelsare applied to a group of predictive inputs in order to determine agroup of corresponding cross-model predictive outputs. For example, if aparticular predictive model of two or more predictive models has a 40%selection probability score, the noted selection probability score mayindicate a preference that a cross-model predictive inference configuredto apply the two or more predictive inputs for a group of predictiveinputs selects the particular predictive models 40% of the times (e.g.,apply the particular predictive model to 40% of the group of predictiveinputs).

The term “cross-model normalization” refers to a computer-implementedprocess for determining a cross-model predictive score based on amodel-specific predictive score determined by a corresponding predictivemodel. For example, if a particular predictive model of two or morepredictive models is configured to determine one model-specificpredictive score that is selected from the model-specific range {A, B,C, D, E, F} organized in a descending order of value, and further if across-model predictive inference associated with the two or morepredictive models is configured to determine cross-model predictiveoutputs based on cross-model predictive scores selected from thecross-model range {0%-100%), performing a cross-model normalizationassociated with the particular predictive model may include operationsconfigured to convert a model-specific predictive output indicating “A”to a cross-model predictive score indicating 100%, a model-specificpredictive output indicating “B” to a cross-model predictive scoreindicating 80%, a model-specific predictive output indicating “C” to across-model predictive score indicating 60%, a model-specific predictiveoutput indicating “D” to a cross-model predictive score indicating 40%,a model-specific predictive output indicating “E” to a cross-modelpredictive score indicating 20%, and a model-specific predictive outputindicating “F” to a cross-model predictive score indicating 0%. Asanother example, if a particular predictive model of two or morepredictive models is configured to determine two model-specificpredictive outputs but a cross-model predictive inference associatedwith the two or more predictive models is configured to determine across-model predictive output based on a single cross-model predictivescores, performing a cross-model normalization associated with theparticular predictive model may include operations configured to combine(e.g., average) the two model-specific predictive outputs to determinethe single cross-model predictive output.

The term “cross-model normalization data” refers to a collection of oneor more data items that describe operations and/or parameters used toperform cross-model normalization for a particular predictive model. Insome embodiments, cross-model normalization data for a particularpredictive model may indicate that the cross-model predictive score foreach model-specific predictive score is determined based on an ascendingor descending order of the model-specific predictive scores. Forexample, given the cross-model predictive scores 500, 200, 100, 50, and5 for predictive inputs A-E respectively, the cross-model normalizationdata may indicate that the cross-model predictive for predictive input C(i.e., with the model-specific predictive score 100) should be 60^(th)percentile since the model-specific predictive score for the predictiveinput C is the third highest model-specific predictive score among themodel-specific predictive scores for the predictive inputs A-E.

The term “cross-model predictive score generation” refers to acomputer-implemented process for performing a cross-model predictiveinference on a particular predictive input. For example, performing across-model predictive score generation may include performingoperations configured to generate a cross-model predictive score for apredictive input based on one or more model-specific predictive scoresfor the predictive input determined by a selected predictive inputassociated with the predictive model.

The term “weighted random selection” refers to a collection of one ormore data items that indicate operations for selecting a particularpredictive model of two or more predictive models to apply to apredictive input of two or more predictive inputs, where the selectionis performed based on a random process defined by one or more randomselection parameters. The weighted random selection is deemed “weighted”because the random selection process is performed based on weightsdefined by the one or more random selection parameters. For example, aweighted random selection may define operations for selecting aparticular predictive model of two or more predictive models based on arandom selection, where the random selection is defined by randomselection parameters that are in turn defined by each selectionprobability score for a predictive model as defined by the modelselection probability distribution for the two or more predictivemodels.

The term “random selection parameter” refers to a collection of one ormore data items that indicate at least one aspect of a probabilitydistribution utilized to perform a weighted random selection. In someembodiments, at least one random selection parameter may be defined byat least one selection probability score for a predictive model. In someembodiments, at least one random selection parameter may be defined byat least one property of at least one predictive input. In someembodiments, at least one random selection parameter may be defined by arandom number generation process.

The term “selected predictive model” refers to a predictive modelselected to be applied to a particular predictive input as part ofcross-model predictive score generation for the particular predictiveinput. For example, the selected predictive model for a particularpredictive input may be defined based on one or more of at least oneweighted selection random parameter for the particular predictive inputand at least one selection probability score.

The term “unselected predictive model” refers to any predictive modelnot selected to be applied to a particular predictive input as part ofcross-model predictive score generation for the particular predictiveinput. For example, if a cross-model predictive inference is associatedwith two predictive models A and B and the predictive model B isselected for application to a particular predictive input, then thepredictive B is an unselected predictive model for the particularpredictive input.

The term “model-specific predictive score” refers to a collection of oneor more data items determined by a corresponding predictive model viaapplying the corresponding predictive model to a particular predictiveinput. For example, a particular predictive model may be configured toprocess particular input properties for a particular merchant identifierto determine an alphabetical merchant promotional outreach interestprediction, where the alphabetical merchant promotional outreachinterest prediction is a model-specific predictive score.

The term “cross-model predictive score” refers to a collection of one ormore data items determined by performing cross-model normalization onone or more corresponding model-specific predictive scores. For example,performing a particular cross-model normalization for a particularpredictive model may include transforming alphabetical merchantpromotional outreach interest predictions generated by the particularpredictive model into numeric merchant promotional outreach interestpredictions, where the alphabetical merchant promotional outreachinterest predictions are a model-specific predictive scores and thenumeric merchant promotional outreach interest predictions arecross-model predictive scores. As another example, performingcross-model normalization for a particular predictive model may includecombining two model-specific predictive scores determined by theparticular predictive model to determine a single cross-model predictivescore for the particular predictive model.

The term “champion predictive model” refers to a predictive modelassociated with a cross-model predictive inference that is designated tobe a superior, a preferred, and/or a primary predictive model for thecross-model predictive inference. For example, a cross-model predictiveinference associated with two or more predictive models may include aneural network predictive model of the two or more predictive modelsthat is designated as the champion predictive model. In someembodiments, a champion predictive model is a predictive model havingone or more of the following properties: a higher correlationcoefficient than the correlation coefficient of the challengerpredictive model, a higher Spearman correlation coefficient than theSpearman correlation coefficient of the challenger predictive model, alower root-mean-square error (RMSE) than the RMSE of the challengerpredictive model, and a lower P-value than the P-value of the challengerpredictive model. In some embodiments, at least one of the correlationcoefficient of a predictive model, a Spearman coefficient of apredictive model, and a RMSE of a predictive model may be determinedbased on the predicted performance of a group of cross-model predictiveoutputs and the actual performance of the group of cross-modelpredictive outputs.

The term “challenger predictive model” refers to any predictive modelassociated with a cross-model predictive inference that is notdesignated as a champion predictive model. For example, a cross-modelpredictive inference associated with three predictive models including aneural network predictive model, a regression predictive model, and aBayesian inference predictive model may designate the neural networkpredictive model as the champion predictive model, and the regressionpredictive model and the Bayesian inference predictive model as thechallenger predictive models.

The term “initial ranking predictive score” refers to a collection ofone or more data items that indicate, for each predictive input of twoor more predictive inputs, a score configured to indicate a position ofthe predictive input in a predictive score generation order for the twoor more predictive inputs. For example, the initial ranking predictivescore for a particular predictive input may indicate that the predictiveinput of two or more predictive input should be the nth predictive inputin a predictive score generation order for the two or more predictiveinputs.

The term “predictive score generation order” refers to a collection ofone or more data items that indicate, for a cross-model predictiveinference of two or more predictive inputs, an order of performingcross-model score generations for each of the two or more predictiveinputs. For example, a particular predictive score generation order fora cross-model predictive inference associated with the predictive inputs{A, B, C, D} may indicate that the cross-model generation for predictiveinput B must performed first, the cross-model generation for predictiveinput A must performed second, the cross-model generation for predictiveinput D must performed third, and the cross-model generation forpredictive input C must performed fourth. The predictive scoregeneration order may be based on a random ordering of predictive inputs.

The term “retrospective predictive success score” refers to a collectionof one or more data items that indicate, for a corresponding predictivemodel, a corresponding prediction-outcome correspondence for thepredictive model between one or more retrospective predictive scoresgenerated by the predictive model and one or more retrospectiveground-truth predictive outcomes corresponding to the one or moreretrospective past predictive score. For example, the retrospectivepredictive success score for a particular predictive model may indicatethat the particular predictive model has accurately target predictionproperties for 50% of past predictive inputs.

The term “retrospective prediction-outcome correspondence” refers to acollection of one or more data items that indicate, for a correspondingpredictive model, a level of correspondence between one or moreretrospective predictive scores generated by the correspondingpredictive model and one or more retrospective ground-truth predictiveoutcomes corresponding to the one or more retrospective predictivescores. For example, a retrospective prediction-outcome correspondencefor a predictive model may indicate that 50% of retrospective predictivescores generated by the predictive model correspond to ground-truthpredictive outcomes corresponding to the noted retrospective predictivescores. In some embodiments, a predictive output value may be said tocorrespond to a corresponding ground-truth value if the similarityand/or closeness between the predictive output value and theground-truth value exceeds a similarity threshold and/or a closenessthreshold (e.g., if the two values are the same).

The term “retrospective predictive score” refers to a cross-modelpredictive score generated by applying a predictive model (e.g., at aprevious time) to a particular predictive input. For example, byapplying a predictive model to a predictive input associated with amerchant identifier, the predictive model may have previously determineda merchant promotional outreach interest prediction of 60%, which may bea retrospective predictive score for the merchant identifier.

The term “retrospective ground-truth predictive outcome” refers to acollection of one or more data items that indicate, for a correspondingretrospective predictive score associated with a predictive input, apredictive score designated as an empirical and/or observed predictivescore for the predictive input. For example, by applying a predictivemodel to a predictive input associated with a merchant identifier, thepredictive model may have previously determined a merchant promotionaloutreach interest prediction of 60%, which may be a retrospectivepredictive score for the merchant identifier. In that example, if theretrospective ground-truth predictive outcome for the retrospectivepredictive score is 100%, then a comparison of the retrospectivepredictive score for the merchant identifier determined by thepredictive model and the retrospective ground-truth predictive outcomefor the retrospective predictive score may indicate a lack ofcorrespondence between the retrospective predictive score and thecorresponding retrospective ground-truth predictive outcome.

The term “cross-model verification” refers to a computer-implementedprocess for determining a level of accuracy of a prediction generated byan unselected predictive model based on the predictive input. Forexample, cross-model verification for a predictive model may indicatethat the predictive model has a 70% accuracy rate for predictionsgenerated by the unselected predictive model based on the predictiveinput.

The term “verification predictive score” refers to a cross-modelpredictive score generated by an unselected predictive model associatedwith a predictive input by applying a predictive model to the predictiveinput. For example, performing cross-model verification may require thatan unselected predictive model associated with a predictive inputdetermine a verification predictive score for the predictive input.

The term “threshold prospective performance time interval” refers to acollection of one or more data items that indicate a time period afterdetermination of a cross-model predictive output for a predictive inputby a selected predictive model for the predictive input, where afterexpiration of the noted time period the system may obtain a prospectiveground-truth success outcome for the predictive input and determine aprospective predictive success score for the predictive input. Forexample, the threshold prospective performance time interval for aparticular predictive input of two or more predictive inputs associatedwith two or more merchant identifiers may be the estimated time periodfor promotional outreach to the two or more merchant identifiers. Thethreshold prospective performance time interval may be determined usinga backtest predictive model that is configured to generate an optimalthreshold perspective performance time interval based on historicaldata.

The term “prospective ground-truth success outcome” refers to acollection of one or more data items that indicate, for a predictiveinput, an empirical and/or observed prediction for the predictive inputafter obtaining empirical and/or observed data for the predictive inputduring the threshold prospective performance time interval associatedwith the predictive input. For example, if the predictive input relatesto a merchant identifier and the prediction relates to a merchantpromotional outreach interest prediction for the merchant identifier,the prospective ground-truth success outcome for the prediction maydescribe an observed merchant promotional outreach interest predictionbased on merchant outreach during a particular time period, wherein thetime period may be the threshold prospective performance time intervalfor the prediction.

The term “prospective predictive success score” refers to a collectionof one or more data items that indicate, for a predictive model, a levelof accuracy of a prediction generated by the unselected predictive modelbased on the predictive input. For example, the prospective predictivesuccess score for a selected predictive model associated with apredictive input may be determined based on the prospective ground-truthsuccess outcome for the predictive input and the cross-model predictivescore for the predictive input. As another example, the prospectivepredictive success score for an unselected predictive model associatedwith a predictive input may be determined based on a recent ground-truthsuccess outcome for the predictive input and the verification predictivescore associated with the unselected predictive model.

The term “performance metric” refers to a collection of one or more dataitems that describe one or more expected and/or desired performanceobjectives for a computing device. For example, a performance metric fora computing device may specify a particular end-user response speedrange for the computing device. As another example, a performance metricfor a computing device may specify a particular processing load rangefor the computing device. As yet another example, a performance metricfor a computing device may specify a particular storage load range forthe computing device.

The term “end-user response speed” refers to a collection of one or moreitems that indicate a speed of processing one or more end-user requestsby a computing device to generate one or more corresponding end-useroutputs. For example, the end-user response speed for a computing devicemay indicate speed of processing end-user prediction requests by thecomputing device.

The term “cross-model range conversion operations” refers to acollection of one or more data items that define operations fortransforming a range of at least one model-specific predictive scoredetermined by the predictive model form a model-specific range to across-model range. For example, cross-model range operations may includeoperations for transforming integer numeric values to percentile values.

The term “model-specific range” refers to a collection of one or moredata items that define a range of a model-specific predictive scoredetermined by a predictive model. Examples of model-specific rangesinclude unbounded numeric ranges (e.g., all positive values), boundednumeric values (e.g., all positive values between 1 and 20),alphabetical values, etc.

The term “cross-model range” refers to a collection of one or more dataitems that define a range of a cross-model predictive score determinedby a predictive model. Examples of cross-model ranges include percentileranges (e.g., all real values between 0% and 100%).

The term “predictive ranking” refers to a collection of one or more dataitems that indicate, for one or more predictive inputs, a ranking of theone or more predictive inputs based on the cross-model predictive scoresfor the one or more predictive inputs. For example, for three predictiveinputs A,B,C having cross-model predictive scores 10%, 70%, and 20%respectively, the predictive ranking of the three predictive inputs maybe as follows: B, C, A.

The term “candidate merchant identifier” refers to a collection of oneor more data items that uniquely identify a provider of goods and/orservices. For example, a candidate merchant identifier may uniquelyidentify a provider of goods and/or services that can be a potentialtarget of and/or is deemed a potential target of a merchant promotionaloutreach.

The term “candidate merchant data structure” refers to a collection ofone or more data items that contain digital information about one ormore properties of one or more providers each uniquely identified by acorresponding candidate merchant identifier. For example, the candidatemerchant data structure for a candidate merchant identifier may containdigital information about one or more of a promotional outreachresponsiveness of a provider uniquely identified by the correspondingcandidate merchant identifier, revenue profile of a provider uniquelyidentified by the corresponding candidate merchant identifier,operational profile of a provider uniquely identified by thecorresponding candidate merchant identifier, promotional expenditures ofa provider uniquely identified by the corresponding candidate merchantidentifier, etc. For example, candidate merchant data structures mayinclude information related to one or more of:

-   number of merchants that have live deals-   number of merchants that have featured deals-   number of merchants that have transactions-   sum of the number of available days-   sum of the number of featured days from each merchant-   sum of the number of transactions from each merchant-   avg_all_prior-   avg_same_service_prior-   avg_percentile_prior-   avg_svc_percentile_prior-   prior_campaigns-   top_merchant_campaign-   this_year_prior_six_pct-   last_year_next_six_pct-   pop_service-   pop_merchant_division_permalink-   pop_service_header-   comp_bookings-   first_comp_tbtw-   last_comp_tbtw-   tbtwrun-   roll12mo_users-   research_ranking-   lead_source-   subs_within5-   distance_to_division_center-   hyperlocal_rank-   n_locations-   online_review_n_reviews-   social_media_fans-   search_engine_n_reviews-   search_engine_rating_nrm-   online_review_rating-   ta_rating-   online_review_price-   subdivision_pg-   units_avail-   this_year_prior_six-   last_year_next_six-   deal_month-   quarter-   julianx

The term “merchant promotional outreach interest prediction” refers to acollection of one or more data items that indicate, for a candidatemerchant identifier, a predicted and/or estimated likelihood that apromotional outreach to the merchant identifier would be successful. Forexample, the merchant promotional outreach interest for a particularmerchant identifier may indicate a promotional outreach to a providerassociated with the merchant identifier has a 70% likelihood of success.

The term “interested merchant identifier” refers to a particularcandidate merchant identifier whose corresponding merchant promotionaloutreach interest prediction indicates a sufficiently high predictedand/or estimated likelihood that a promotional outreach to theparticular candidate merchant would be successful. For example, aninterested merchant may be a merchant identifier whose correspondingmerchant promotional outreach interest prediction exceeds a threshold.As another example, an interested merchant may be a merchant of two ormore candidate whose corresponding merchant promotional outreachinterest prediction is among the top n merchant promotional outreachinterests for the two or more candidate merchant identifiers.

The term “randomly-selected predictive model” refers to any predictivemodel selected by a weighted random selection which has not yet beendesignated a selected predictive model for a predictive input. Forexample, if two or more predictive models include a neural networkpredictive model and a Bayesian inference network predictive model, aweighted random selection may select the Bayesian inference predictivemodel, in which case the Bayesian inference predictive model may be arandomly-selected predictive model.

The term “candidate model-specific predictive score” refers to amodel-specific predictive score generated by a randomly-selectedpredictive model. For example, if a Bayesian inference predictive modelhas been selected by a weighted random selection but has not yet beendesignated as a selected predictive model for a predictive input, amodel-specific predictive score generated by the Bayesian inferencepredictive model may be a candidate model-specific predictive score.

The term “candidate cross-model predictive score” refers to across-model predictive score generated by a randomly-selected predictivemodel. For example, if a Bayesian inference predictive model has beenselected by a weighted random selection but has not yet been designatedas a selected predictive model for a predictive input, a cross-modelpredictive score generated by the Bayesian inference predictive modelmay be a candidate cross-model predictive score.

The term “candidate cross-model score adoption data” refers to acollection of one or more data items that indicate requirements forwhether a randomly-selected predictive model should be adopted as aselected predictive model for a predictive based on at least onecandidate cross-model predictive score generated by a randomly-selectedpredictive model. For example, candidate cross-model score adoption datamay indicate that a randomly-selected predictive model should bedesignated as a selected predictive model for a predictive input if eachcandidate model-specific predictive score associated with the predictivemodel has a threshold degree of accuracy.

Example System Architecture for Implementing Embodiments of the PresentDisclosure

Methods, apparatuses, and computer program products of the presentdisclosure may be embodied by any of a variety of devices. For example,the method, apparatus, and computer program product of an exampleembodiment may be embodied by a networked device (e.g., an enterpriseplatform), such as a server or other network entity, configured tocommunicate with one or more devices, such as one or more clientcomputing devices. Additionally or alternatively, the computing devicemay include fixed computing devices, such as a personal computer or acomputer workstation. Still further, example embodiments may be embodiedby any of a variety of mobile devices, such as a portable digitalassistant (PDA), mobile telephone, smartphone, laptop computer, tabletcomputer, wearable, or any combination of the aforementioned devices.

FIG. 1 illustrates an example system architecture 100 within whichembodiments of the present disclosure may operate. The architecture 100includes one or more client computing devices, such as a clientcomputing device 101, that interact with a predictive analysis system102 over a communication network 103 to transmit predictive analysisrequests. In response to receiving a predictive analysis request from aclient computing device 101, the predictive analysis system 102 mayperform a predictive analysis task corresponding to the predictiveanalysis request to generate a predictive analysis output correspondingto the predictive analysis request and the predictive analysis task. Thepredictive analysis system 102 may then transmit the generatedpredictive analysis output to the client computing device 101.

For example, the client computing device 101 may be a computing deviceassociated with a promotional agent user profile which transmits (overthe communication network 103) predictive analysis requests seekingmerchant promotional outreach interest predictions associated withparticular candidate merchant identifiers. In response, the predictiveanalysis system 102 may perform predictive analysis tasks configured todetermine the merchant promotional outreach interest predictions inorder to generate, for each candidate merchant identifier, a merchantpromotional outreach interest prediction, which may be a predictiveanalysis output. The predictive analysis system 102 may then provide thegenerated merchant promotional outreach interest predictions to theclient computing device 101.

As another example, the client computing device 101 may be a computingdevice associated with a promotional agent user profile which transmits(over the communication network 103) predictive analysis requestsseeking consumer promotional outreach interest predictions associatedwith particular candidate consumers. In response, the predictiveanalysis system 102 may perform predictive analysis tasks configured todetermine the consumer promotional outreach interest predictions inorder to generate, for each candidate consumer, a consumer promotionaloutreach interest prediction, which may be a predictive analysis output.The predictive analysis system 102 may then provide the generatedconsumer promotional outreach interest predictions to the clientcomputing device 101.

As yet another example, the client computing device 101 may be acomputing device associated with a healthcare professional whichtransmits (over the communication network 103) predictive analysisrequests seeking patient health predictions associated with particularcandidate patients. In response, the predictive analysis system 102 mayperform predictive analysis tasks configured to determine the patienthealth predictions in order to generate, for each candidate patient, apatient health prediction, which may be a predictive analysis output.The predictive analysis system 102 may then provide the generatedpatient health predictions to the client computing device 101.

Communication network 103 may include any wired or wirelesscommunication network including, for example, a wired or wireless localarea network (LAN), personal area network (PAN), metropolitan areanetwork (MAN), wide area network (WAN), or the like, as well as anyhardware, software and/or firmware required to implement it (such as,e.g., network routers, etc.). For example, the communication network 103may include a cellular telephone, a 902.11, 902.16, 902.20, and/or WiMaxnetwork. Further, the communication network 103 may include a publicnetwork, such as the Internet, a private network, such as an intranet,or combinations thereof, and may utilize a variety of networkingprotocols now available or later developed including, but not limited toTCP/IP based networking protocols. For instance, the networking protocolmay be customized to suit the needs of the predictive analysis server111. In one embodiment, the protocol is a custom protocol of JSONobjects sent via a Websocket channel. In one embodiment, the protocol isJSON over RPC, JSON over REST/HTTP, and the like.

A client computing device 101 may be any computing device as definedbelow. Electronic data received by the predictive analysis server 111from the client computing devices 101 may be provided in various formsand via various methods. For example, the client computing devices 101may include desktop computers, laptop computers, smartphones, netbooks,tablet computers, wearables, and the like. An example architecture for aclient computing device 101 is depicted in the apparatus 300 of FIG. 3 .

In embodiments where a client computing device 101 is a mobile device,such as a smart phone or tablet, the client computing device 101 mayexecute an “app” to interact with the predictive analysis server 111.Such apps are typically designed to execute on mobile devices, such astablets or smartphones. For example, an app may be provided thatexecutes on mobile device operating systems such as iOS®, Android®, orWindows®. These platforms typically provide frameworks that allow appsto communicate with one another and with particular hardware andsoftware components of mobile devices. For example, the mobile operatingsystems named above each provide frameworks for interacting withlocation services circuitry, wired and wireless network interfaces, usercontacts, and other applications. Communication with hardware andsoftware modules executing outside of the app is typically provided viaapplication programming interfaces (APIs) provided by the mobile deviceoperating system. Additionally or alternatively, the client computingdevice 101 may interact with the predictive analysis server 111 via aweb browser. As yet another example, the client computing device 101 mayinclude various hardware or firmware designed to interface with thepredictive analysis server 111.

The predictive analysis system 102 may include a predictive analysisserver 111 and a predictive model repository 112. The predictiveanalysis server 111 may be configured to perform predictive analysistasks and generate predictive analysis outputs using multiple predictivemodels, where each predictive model is characterized by one or moreparameters. The predictive model repository 112 is configured to storeparameters characterizing the multiple predictive models as well ascross-model normalization data for each predictive model of the multiplepredictive models. For example, the predictive model repository 112 maystore weights and/or biases characterizing a neural network predictivemodel, reinforcement learning parameters characterizing a reinforcementlearning predictive model, evolutionary parameters characterizing anevolutionary learning predictive model, etc. In an embodiment, thepredictive analysis server 111 can generate one or more correlationcoefficients characterizing the multiple predictive models based on acorrelation statistical measure of a degree of association betweenvariables of the multiple predictive models. Furthermore, the predictivemodel repository 112 can be configured to store one or more correlationcoefficients characterizing the multiple predictive models. Additionallyor alternatively, in an embodiment, the predictive analysis server 111can generate one or more area under the curve parameters characterizingthe multiple predictive models based on an area under the curvestatistical measure of optimal predictive models. Furthermore, thepredictive model repository 112 can be configured to store one or morearea under the curve parameters characterizing the multiple predictivemodels. In certain embodiments, the predictive analysis server 111 canrepeatedly analyze and/or generate the parameters characterizing themultiple predictive models and/or the cross-model normalization data foreach predictive model. For example, the predictive analysis server 111can analyze the parameters characterizing the multiple predictive modelsand/or the cross-model normalization data for each predictive model on adaily basis. Based on the analysis of the parameters characterizing themultiple predictive models and/or the cross-model normalization data foreach predictive model, the predictive analysis server 111 can determineone or more modified parameters and/or modified cross-modelnormalization data for the predictive models. In an embodiment, thepredictive analysis server 111 can generate summary statistics for eachpredictive model based on the analysis of the parameters characterizingthe multiple predictive models and/or the cross-model normalization datafor each predictive model.

The predictive analysis server 111 may be embodied as a computer orcomputers. An example architecture for the predictive analysis server111 is depicted in the apparatus 200 of FIG. 2 . The predictive analysisserver 111 may provide for sending electronic data and/or receivingelectronic data from various sources, including but not limited to theclient computing devices 101. For example, the predictive analysisserver 111 may receive predictive analysis requests from a clientcomputing device 101 and transmit corresponding predictive analysisoutputs to the client computing device 101. In some embodiments, thepredictive analysis server 111 may be configured to perform cross-modelpredictive inference, cross-model predictive score generation,cross-model score verification, and/or cross-model normalization, asfurther described below.

An example architecture for the predictive analysis server 111 isdepicted in the apparatus 200 of FIG. 2 . As depicted in FIG. 2 , theapparatus 200 includes processor 202, memory 204, input/output circuitry206, communications circuitry 208, and predictive analysis circuitry210. The apparatus 200 may be configured to execute the operationsdescribed herein with respect to FIGS. 4-7 . Although these components202-210 are described with respect to functional limitations, it shouldbe understood that the particular implementations necessarily includethe use of particular hardware. It should also be understood thatcertain of these components 202-210 may include similar or commonhardware. For example, two sets of circuitries may both leverage use ofthe same processor, network interface, storage medium, or the like toperform their associated functions, such that duplicate hardware is notrequired for each set of circuitries.

In one embodiment, the processor 202 (and/or co-processor or any otherprocessing circuitry assisting or otherwise associated with theprocessor) may be in communication with the memory 204 via a bus forpassing information among components of the apparatus. The memory 204 isnon-transitory and may include, for example, one or more volatile and/ornon-volatile memories. In other words, for example, the memory 204 maybe an electronic storage device (e.g., a computer-readable storagemedium). The memory 204 may be configured to store information, data,content, applications, instructions, or the like for enabling theapparatus to carry out various functions in accordance with exampleembodiments of the present disclosure.

The processor 202 may be embodied in a number of different ways and may,for example, include one or more processing devices configured toperform independently. In some preferred and non-limiting embodiments,the processor 202 may include one or more processors configured intandem via a bus to enable independent execution of instructions,pipelining, and/or multithreading. The use of the term “processingcircuitry” may be understood to include a single core processor, amulti-core processor, multiple processors internal to the apparatus,and/or remote or “cloud” processors.

In some preferred and non-limiting embodiments, the processor 202 may beconfigured to execute instructions stored in the memory 204 or otherwiseaccessible to the processor 202. In some preferred and non-limitingembodiments, the processor 202 may be configured to execute hard-codedfunctionalities. As such, whether configured by hardware or softwaremethods, or by a combination thereof, the processor 202 may represent anentity (e.g., physically embodied in circuitry) capable of performingoperations according to an embodiment of the present disclosure whileconfigured accordingly. Alternatively, as another example, when theprocessor 202 is embodied as an executor of software instructions, theinstructions may specifically configure the processor 202 to perform thealgorithms and/or operations described herein when the instructions areexecuted.

As an example, the processor 202 may be configured to maintain one ormore communication channels connecting a plurality of client computingdevices 101 to enable message sharing/dissemination therebetween. Theprocessor 202 ensures that messages intended for exchange between theclient computing devices 101 within the particular communication channelare properly disseminated to those client computing devices 101 fordisplay within respective display windows provided via the clientcomputing devices 101.

Moreover, the processor 202 may be configured to synchronize messagesexchanged on a particular communication channel with a database forstorage and/or indexing of messages therein. In certain embodiments, theprocessor 202 may provide stored and/or indexed messages to theinterface computing entity 109 for dissemination to client computingdevices 101.

In one embodiment, the apparatus 200 may include input/output circuitry206 that may, in turn, be in communication with processor 202 to provideoutput to the user and, in one embodiment, to receive an indication of auser input. The input/output circuitry 206 may comprise a user interfaceand may include a display, and may comprise a web user interface, amobile application, a client computing device, a kiosk, or the like. Inone embodiment, the input/output circuitry 206 may also include akeyboard, a mouse, a joystick, a touch screen, touch areas, soft keys, amicrophone, a speaker, or other input/output mechanisms. The processorand/or user interface circuitry comprising the processor may beconfigured to control one or more functions of one or more userinterface elements through computer program instructions (e.g., softwareand/or firmware) stored on a memory accessible to the processor (e.g.,memory 204, and/or the like).

The communications circuitry 208 may be any means such as a device orcircuitry embodied in either hardware or a combination of hardware andsoftware that is configured to receive and/or transmit data from/to anetwork and/or any other device, circuitry, or module in communicationwith the apparatus 200. In this regard, the communications circuitry 208may include, for example, a network interface for enablingcommunications with a wired or wireless communication network.

For example, the communications circuitry 208 may include one or morenetwork interface cards, antennae, buses, switches, routers, modems, andsupporting hardware and/or software, or any other device suitable forenabling communications via a network. Additionally or alternatively,the communications circuitry 208 may include the circuitry forinteracting with the antenna/antennae to cause transmission of signalsvia the antenna/antennae or to handle receipt of signals received viathe antenna/antennae.

Predictive analysis circuitry 210 includes hardware configured toexecute operations configured to perform cross-model predictiveinference, cross-model predictive score generation, cross-model scoreverification, and/or cross-model normalization, as further describedbelow. The predictive analysis circuitry 210 may utilize processingcircuitry, such as the processor 202, to perform these actions. However,it should also be appreciated that, In one embodiment, the predictiveanalysis circuitry 210 may include a separate processor, speciallyconfigured Field Programmable Gate Array (FPGA), or Application SpecificIntegrated Circuit (ASIC) for performing the functions described herein.The predictive analysis circuitry 210 may be implemented using hardwarecomponents of the apparatus configured by either hardware or softwarefor implementing these planned functions.

An example architecture for a client computing device 101 is depicted inthe apparatus 300 of FIG. 3 . As depicted in FIG. 3 , the apparatus 300includes processor 301, memory 303, input/output circuitry 305, andcommunications circuitry 307. Although these components 301-307 aredescribed with respect to functional limitations, it should beunderstood that the particular implementations necessarily include theuse of particular hardware. It should also be understood that certain ofthese components 301-307 may include similar or common hardware. Forexample, two sets of circuitries may both leverage use of the sameprocessor, network interface, storage medium, or the like to performtheir associated functions, such that duplicate hardware is not requiredfor each set of circuitries.

In one embodiment, the processor 301 (and/or co-processor or any otherprocessing circuitry assisting or otherwise associated with theprocessor) may be in communication with the memory 303 via a bus forpassing information among components of the apparatus. The memory 303 isnon-transitory and may include, for example, one or more volatile and/ornon-volatile memories. In other words, for example, the memory 303 maybe an electronic storage device (e.g., a computer-readable storagemedium). The memory 303 may be configured to store information, data,content, applications, instructions, or the like for enabling theapparatus 300 to carry out various functions in accordance with exampleembodiments of the present disclosure. For example, the memory 303 maybe configured to cache messages exchanged on one or more predictiveanalysis, such that the processor 301 may provide various messages toclient computing devices (e.g., on an as needed or as requested basis).

The processor 301 may be embodied in a number of different ways and may,for example, include one or more processing devices configured toperform independently. In some preferred and non-limiting embodiments,the processor 301 may include one or more processors configured intandem via a bus to enable independent execution of instructions,pipelining, and/or multithreading.

In some preferred and non-limiting embodiments, the processor 301 may beconfigured to execute instructions stored in the memory 303 or otherwiseaccessible to the processor 301. In some preferred and non-limitingembodiments, the processor 301 may be configured to execute hard-codedfunctionalities. As such, whether configured by hardware or softwaremethods, or by a combination thereof, the processor 301 may represent anentity (e.g., physically embodied in circuitry) capable of performingoperations according to an embodiment of the present disclosure whileconfigured accordingly. Alternatively, as another example, when theprocessor 301 is embodied as an executor of software instructions, theinstructions may specifically configure the processor 301 to perform thealgorithms and/or operations described herein when the instructions areexecuted.

In one embodiment, the apparatus 300 may include input/output circuitry305 that may, in turn, be in communication with processor 301 to provideoutput to the user and, In one embodiment, to receive an indication of auser input. The input/output circuitry 305 may comprise a user interfaceand may include a display, and may comprise a web user interface, amobile application, a client computing device, a kiosk, or the like. Inone embodiment, the input/output circuitry 305 may also include akeyboard, a mouse, a joystick, a touch screen, touch areas, soft keys, amicrophone, a speaker, or other input/output mechanisms.

The communications circuitry 307 may be any means such as a device orcircuitry embodied in either hardware or a combination of hardware andsoftware that is configured to receive and/or transmit data from/to anetwork and/or any other device, circuitry, or module in communicationwith the apparatus 300. In this regard, the communications circuitry 307may include, for example, a network interface for enablingcommunications with a wired or wireless communication network. Forexample, the communications circuitry 307 may include one or morenetwork interface cards, antennae, buses, switches, routers, modems, andsupporting hardware and/or software, or any other device suitable forenabling communications via a network. Additionally or alternatively,the communications circuitry 307 may include the circuitry forinteracting with the antenna/antennae to cause transmission of signalsvia the antenna/antennae or to handle receipt of signals received viathe antenna/antennae.

It is also noted that all or some of the information discussed hereincan be based on data that is received, generated and/or maintained byone or more components of apparatus 300. In one embodiment, one or moreexternal systems (such as a remote cloud computing and/or data storagesystem) may also be leveraged to provide at least some of thefunctionality discussed herein.

The term “circuitry” should be understood broadly to include hardwareand, in one embodiment, software for configuring the hardware. Withrespect to components of each apparatus 200, 300, the term “circuitry”as used herein should therefore be understood to include particularhardware configured to perform the functions associated with theparticular circuitry as described herein. For example, in oneembodiment, “circuitry” may include processing circuitry, storage media,network interfaces, input/output devices, and the like. In oneembodiment, other elements of the apparatus 200 may provide orsupplement the functionality of particular circuitry. For example, theprocessor 202 may provide processing functionality, the memory 204 mayprovide storage functionality, the communications circuitry 208 mayprovide network interface functionality, and the like. Similarly, otherelements of the apparatus 300 may provide or supplement thefunctionality of particular circuitry. For example, the processor 301may provide processing functionality, the memory 303 may provide storagefunctionality, the communications circuitry 307 may provide networkinterface functionality, and the like.

As will be appreciated, any such computer program instructions and/orother type of code may be loaded onto a computer, processor or otherprogrammable apparatus’s circuitry to produce a machine, such that thecomputer, processor or other programmable circuitry that execute thecode on the machine creates the means for implementing variousfunctions, including those described herein.

As described above and as will be appreciated based on this disclosure,embodiments of the present disclosure may be configured as methods,mobile devices, backend network devices, and the like. Accordingly,embodiments may comprise various means including entirely of hardware orany combination of software and hardware. Furthermore, embodiments maytake the form of a computer program product on at least onenon-transitory computer-readable storage medium having computer-readableprogram instructions (e.g., computer software) embodied in the storagemedium. Any suitable computer-readable storage medium may be utilizedincluding non-transitory hard disks, CD-ROMs, flash memory, opticalstorage devices, or magnetic storage devices.

Moreover, although not shown, various embodiments of a predictiveanalysis server 111 may include one or more databases configured forstoring and/or indexing predictive models associated with the predictiveanalysis server 111.

Example Data Flows of Embodiments of the Present Disclosure

Various embodiments of the present disclosure address efficiency andreliability challenges for predictive data analysis systems stemmingfrom predictive model diversity by introducing various techniquesrelated to cross-model predictive inference that enable detectingoptimal predictive models for individual predictive inputs and applyingthe detected optimal predictive models to the noted individualpredictive inputs. Examples of the noted techniques include cross-modelpredictive inference, cross-model predictive score generation,cross-model score verification, and cross-model normalization. In anaspect, various embodiments additionally or alternatively improveperformance of a processor configured to execute one or more operationsassociated with various techniques related to cross-model predictiveinference. For example, various embodiments improve processing speedand/or reduce a number of computational resources associated withvarious techniques related to cross-model predictive inference.Furthermore, various embodiments additionally or alternatively improvetraining of a predictive model associated with cross-model predictiveinference, cross-model predictive score generation, cross-model scoreverification, and/or cross-model normalization. For example, variousembodiments additionally or alternatively provide improved or reducedparameters, improved values, improved weights, and/or improvedthresholds for a predictive model associated with cross-model predictiveinference, cross-model predictive score generation, cross-model scoreverification, and/or cross-model normalization.

For example, by utilizing some aspects of cross-model predictiveinference and some aspects of cross-model predictive score generation,various embodiments of the present disclosure enable selectivelyutilizing multiple predictive models to process multiple predictiveinputs, thus enabling selection of the most optimal predictive model foreach predictive input when performing cross-model predictive inferences.In an embodiment, the predictive inputs can be related to marketingleads for merchants of goods or services. As another example, byutilizing some aspects of cross-model score verification, variousembodiments of the present disclosure enable determining selectionprobabilities of individual predictive models based on empiricalobservations about correspondence of predictions generated by thosemodels and real-world predictions, thus enabling improved detection ofoptimal predictive models for individual predictive inputs whenperforming cross-model predictive inferences. As yet another example, byutilizing some aspects of cross-model normalization, various embodimentsof the present disclosure enable generating normalized outputs having across-model format based on model-specific outputs each having amodel-specific format, thus enabling better integration andinteroperability between various predictive models when performingcross-model predictive inferences.

Cross-Model Predictive Inference

FIG. 4 is an example flow diagram illustrating performing a cross-modelpredictive inference. The process 400 illustrated in FIG. 4 is describedherein with reference to a predictive analysis server, such as thepredictive analysis server 111 of FIG. 1 , but can be performed by anysystem of one or more computers, such as a system that includes thepredictive analysis server 111 of FIG. 1 . Each block of the flowchart,and combinations of blocks in the flowchart, may be implemented byvarious means such as hardware, firmware, circuitry and/or other devicesassociated with execution of software including one or more computerprogram instructions. For example, one or more of the proceduresdescribed in FIG. 4 may be embodied by computer program instructions,which may be stored by a non-transitory memory of an apparatus employingan embodiment of the present disclosure and executed by a processor inthe apparatus. These computer program instructions may direct a computeror other programmable apparatus to function in a particular manner, suchthat the instructions stored in the computer-readable storage memoryproduce an article of manufacture, the execution of which implements thefunction specified in the flowchart block(s).

As depicted in FIG. 4 , the process 400 starts at blocks 401-402 whenthe predictive analysis server 111 identifies a group of predictiveinputs (at block 401) and identifies a group of predictive models (atblock 402). Examples of predictive inputs may include predictive inputsassociated with candidate merchants, candidate patients, candidatetravel paths, etc. For example, in an embodiment, the predictive inputscan be related to data indicative of marketing leads for candidatemerchants of goods or services. In certain embodiments, the dataindicative of the marketing leads can be related to a particular type ofmarket for the goods or services. Examples of predictive models mayinclude predictive models that include one or more of regression-basedmodels, neural-network-based models, Bayesian-inference-based models,etc. In an embodiment, the predictive models can include two or morepredictive models that generate respective rankings of merchants thatprovide goods and/or services. In an aspect, a first predictive modelfrom the predictive models can be different than a second predictivemodel from the predictive models. For instance, the first predictivemodel and the second predictive model can be configured with differentarchitectural complexities and/or different operational requirements. Inone example, the first predictive model and the second predictive modelcan be different types of machine learning models. However, the firstpredictive model and the second predictive model can be trained withcorresponding training data. In certain embodiments, the secondpredictive model can be trained based on data associated with the firstpredictive model. Additionally or alternatively, the first predictivemodel can be trained based on data associated with the second predictivemodel. In an embodiment, a predictive model can be an incrementalitymodel that provides merchant ranking based on incremental value.

In certain embodiments, the predictive inputs can be obtained from acandidate merchant data structure. A merchant can be, for example, apotential target for a merchant promotional outreach and/or anelectronic communication for a merchant promotional outreach. In anaspect, an electronic communication for a merchant promotional outreachcan be transmitted to a consumer device (e.g., a client computing devicefrom the client computing devices 101) to facilitate rendering of dataassociated with the merchant promotion outreach via an electronicinterface (e.g., a graphical user interface) of the consumer device. Incertain embodiments, a ranking of merchants can, for example, rankmerchants based on predicted value of transmitting an electroniccommunication (e.g., an electronic communication for a merchantpromotion outreach) to a consumer device associated with a merchant inorder to maximize efficiency and/or revenue provided by merchants. Forinstance, a ranking of merchants can, for example, rank merchants basedon a predicted incremental value with respect to a given market for agood and/or service. In certain embodiments, a ranking of merchants canbe an optimized list of merchants to facilitate a decrease in a numberof computing resources for transmitting electronic communications to oneor more consumer devices. Accordingly, with the ranking of merchantsprovided by the prediction models, bandwidth of a communication networkassociated with transmission of electronic communications can also beimproved.

At block 403, the predictive analysis server obtains a model selectionprobability distribution that defines, for each predictive modelidentified in block 402, a respective selection probability score. Insome embodiments, the predictive models identified in block 402 includea champion predictive model and a challenger predictive model, and theselection probability score for a champion predictive model exceeds theselection probability score for any predictive challenger model. In someembodiments, the challenger predictive model can be a modified versionof the champion predictive model. For instance, in certain embodiments,one or more operations of the challenger predictive model can bedifferent than one or more operations of the champion predictive model.Additionally or alternatively, one or more parameters of the challengerpredictive model can be different than one or more parameters of thechampion predictive model. Additionally or alternatively, a value of oneor more parameters of the challenger predictive model can be differentthan corresponding parameters of the champion predictive model. Incertain embodiments, the model selection probability distribution can bea collection of one or more data items that indicate a selectionprobability score for the champion predictive model and the challengerpredictive model. For example, the model selection probabilitydistribution may indicate a first selection probability score (e.g., 80%selection probability score) for the champion predictive model and asecond selection probability score (e.g., 20% selection probabilityscore) for the challenger predictive model.

In some embodiments, the selection probability score for at least afirst predictive model of the predictive models identified in block 402is determined based on the retrospective prediction-outcomecorrespondence of the first predictive model compared to theretrospective prediction-outcome correspondences of one or more secondpredictive models identified in block 402. In some embodiments, theselection probability score for a first predictive model is adjustedafter each iteration of performing cross-model score verification forthe identified predictive models, as described below. In someembodiments, the champion or challenger designation of each predictivemodel is adjusted after each iteration of performing cross-model scoreverification for the identified predictive models, as described below.In some embodiments, selection of champion predictive model is based onone or more rules for model selection (e.g., a rule that requires aparticular predictive model for predictive inputs having particularcategory, such as a rule that requires selecting a particular predictivemodel for restaurants).

At block 404, the predictive analysis server 111 obtains cross-modelnormalization data for each predictive model identified in block 402. Insome embodiments, the cross-model normalization data for at least onepredictive model of the predictive models defines one or morecross-model conversion operations for the predictive model. The one ormore cross-model conversion operations for a respective predictive modelcan be configured to convert the cross-model predictive output for therespective predictive model each having an output-specific range to across-model value for the predictive model having a cross-model range.Furthermore, each cross-model predictive score for a predictive modelcan be determined based on a respective cross-model value for thepredictive model. In some embodiments, the cross-model normalizationdata for at least one predictive model converts two or moremodel-specific predictive scores for the predictive model into across-model value for the predictive model. The cross-model value canbe, for example, a prediction for a degree of value (e.g., anincremental value) added to a given market for a good and/or serviceprovided by a merchant.

At block 405, the predictive analysis server 111 determines across-model predictive score for each predictive input identified inblock 401. In some embodiments, the predictive analysis system 102determines a cross-model predictive score for a particular predictiveinput based on at least one of the model selection probabilitydistribution obtained in block 403 and/or each cross-model normalizationdata for a predictive model obtained in block 404. In some embodiments,the predictive analysis server 111 determines for each predictive inputof the plurality of predictive inputs, an initial ranking predictivescore by applying the champion predictive model to the predictive input.In some of those embodiments, determining the cross-model predictivescore for each predictive input of the identified predictive inputsincludes determining each cross-model predictive score in a predictivescore generation order, where the predictive score generation order isdetermined based on each initial ranking predictive score associatedwith a predictive input of the identified predictive inputs. Forexample, if the predictive score generation order indicates that a firstpredictive input has a higher initial ranking predictive score than asecond predictive input, the predictive analysis server 111 maydetermine the cross-model predictive score for the first predictiveinput before determining the cross-model predictive score for the secondpredictive input.

In some embodiments, operations of block 405 with respect to apredictive input may be performed in accordance with the operationsdepicted in various blocks of the process depicted in FIG. 5 . Theprocess depicted in FIG. 5 begins at block 501 when the predictiveanalysis server 111 determines a selected predictive model for thepredictive input. In some embodiments, the predictive analysis server111 determines the selected predictive model for the predictive inputbased on a weighted random selection for the predictive inputcharacterized by one or more random selection parameters for thepredictive input. In some embodiments, at least one random selectionparameter of the one or more random selection parameters is determinedbased on the model selection probability distribution. In someembodiments, the model selection probability is an uniform distributionof the predictive models. For example, in some embodiments, given auniform distribution {40%, 60%} for two predictive models A and Brespectively, the predictive analysis server 111 randomly selectspredictive models for predictive inputs in a manner that causespredictive model A to be selected for 40% of the predictive inputs andpredictive model B to be selected for 60% of the predictive inputs.

In some embodiments, the predictive models identified in block 402include a champion predictive model and a challenger predictive model,and the random selection parameters for a champion predictive model areconfigured to cause the champion predictive model to be selected at ahigher predicted frequency than any challenger predictive model. In someembodiments, at least one random selection parameter for a firstpredictive model identified in block 402 is determined based on theretrospective prediction-outcome correspondence of the first predictivemodel compared to the retrospective prediction-outcome correspondencesof one or more second predictive models identified in block 402. In someembodiments, at least one random selection parameter for at least afirst predictive model of the predictive models is adjusted after eachiteration of performing cross-model score verification, as describedbelow. In some embodiments, the champion or challenger designation ofeach predictive model is adjusted after performing each cross-modelscore verification, as described below. In some embodiments, thecross-model score verification is a backtest cross-model scoreverification (e.g., a cross-model score verification based on historicaldata). In some embodiments, the cross-model score verification is aforward-test cross-model verification (e.g., a cross-model scoreverification based on incoming data). In some embodiments, thecross-model score verification includes both a backtest cross-modelscore verification and a forward-test cross-model score verification.

In certain embodiments, a merchant ranking test (e.g., a merchantacquisition ranking evaluation) can be employed to test whether acorrelation measure between a predicted rank (e.g., a model score rank)and an observed rank (e.g., a financial metric rank) from the championpredictive model and the challenger predictive model satisfies a definedcriterion. For example, the merchant ranking test (e.g., a merchantacquisition ranking evaluation) can be employed to determine a degree ofdifference between the champion predictive model and the challengerpredictive model based on the correlation measure. In an aspect, themerchant ranking test can be based on a primary metric associated withpredicted metrics for new merchants for a first interval of time (e.g.,30 days), a secondary metric associated with observed metrics for newmerchants for the first interval of time (e.g., 30 days), and anauxiliary metric associated with observed metrics for a second intervalof time (e.g., 75 days). In an example, the primary metric can bedetermined for the champion predictive model, the challenger predictivemodel, a difference between the champion predictive model and thechallenger predictive model, a standard deviation between the championpredictive model and the challenger predictive model, a probabilityvalue between the champion predictive model and the challengerpredictive model, and/or a determined significance between the championpredictive model and the challenger predictive model. Furthermore, thesecondary metric can be determined for the champion predictive model,the challenger predictive model, a difference between the championpredictive model and the challenger predictive model, a standarddeviation between the champion predictive model and the challengerpredictive model, a probability value between the champion predictivemodel and the challenger predictive model, and/or a determinedsignificance between the champion predictive model and the challengerpredictive model. The auxiliary metric can also be determined for thechampion predictive model, the challenger predictive model, a differencebetween the champion predictive model and the challenger predictivemodel, a standard deviation between the champion predictive model andthe challenger predictive model, a probability value between thechampion predictive model and the challenger predictive model, and/or adetermined significance between the champion predictive model and thechallenger predictive model.

In certain embodiments, a merchant financial performance distribution(e.g., a metal segment setup) among one or more segments from thechampion predictive model and the challenger predictive model can becompared to determine a degree of difference between the championpredictive model and the challenger predictive model. In an embodiment,at least one parameter of the challenger predictive model can beadjusted based on a degree of difference between the champion predictivemodel and the challenger predictive model. In another embodiment, themerchant financial performance distribution can be calculated at aparticular time between an interval of time for the merchant financialperformance distribution. For example, the merchant financialperformance distribution can be calculated on a 15^(th) day of a monthover all scored merchants to represent merchant financial performancethroughout the month. In certain embodiments, the merchant financialperformance distribution can employ a median value as a statisticalmeasure for comparison. In certain embodiments, quintile analysis can beperformed with respect to the merchant financial performancedistribution. For example, the merchant financial performancedistribution can be divided into multiple segments (e.g. five segments)that respectively correspond to 20% segments of the merchant financialperformance distribution. As such, differences between the championpredictive model and the challenger predictive model can be comparedbased on subsets portions of the merchant financial performancedistribution. In certain embodiments, a heatmap can be employed toanalyze differences between the champion predictive model and thechallenger predictive model. For example, the heatmap can provide avisualization of differences between different portions of the championpredictive model and the challenger predictive model. In certainembodiments, a distribution density plot can be employed to analyzedifferences between the champion predictive model and the challengerpredictive model. For example, the distribution density plot can providea visualization of differences between different portions of thechampion predictive model and the challenger predictive model.

In some embodiments, the champion predictive model and the challengerpredictive model can be tested in parallel. For example, generationand/or verification of scores for the champion predictive model and thechallenger predictive model can be simultaneously performed. In anembodiment, a first score can be generated for the champion predictivemodel and a second score can be generated for the challenger predictivemodel. The first score and the second score can be generated in parallel(e.g., the first score and the second score can be simultaneouslygenerated). Furthermore, the second score for the challenger predictivemodel can be compared to the first score for the champion predictivemodel.

In another embodiment, scoring of the champion predictive model and thechallenger predictive model can be evaluated based on two or moreobjective functions. An objective function can be a set of constraintsrelated to one or more metrics for one or more merchants. Furthermore,an objective function can be related to one or more goals related to oneor more merchants. In an aspect, an objective function can include afirst constraint related to a first metric, a second constraint relatedto a second metric, etc. In another aspect, a metric can include, but isnot limited to, information related to a geographical location, purchasebehavior of a consumer, revenue for a particular market for a good orservice, a type of good or service, etc. In an example, scoring of thechampion predictive model and the challenger predictive model can beevaluated based on a first objective function (e.g., a purchasefrequency for a particular good or service), a second objective function(e.g., incremental revenue at a particular geographical location), and athird objective function (e.g., revenue at multiple geographicallocations). In an embodiment, the second score for the challengerpredictive model can be compared to the first score for the championpredictive model based on the first objective function (e.g., thepurchase frequency for a particular good or service), the secondobjective function (e.g., the incremental revenue at the particulargeographical location), and the third objective function (e.g., therevenue at the multiple geographical locations). As such, differentmodel parameters between the champion predictive model and thechallenger predictive model can be tested and/or evaluated. In certainembodiment, the champion predictive model or the challenger predictivemodel can be selected based on the evaluation of the score for thechampion predictive model and the respective scores for the challengerpredictive model.

In some embodiments, at least one random selection parameter may bedefined by at least one selection probability score for a predictivemodel. In some embodiments, at least one random selection parameter maybe defined by at least one property of at least one predictive input. Insome embodiments, at least one random selection parameter may be definedby a random number generation process.

In some embodiments, operations of block 501 may be performed inaccordance with the operations depicted in various blocks of the processdepicted in FIG. 6 . The process depicted in FIG. 6 begins at block 601when the predictive analysis server 111 generates a firstrandomly-selected predictive model for the predictive input identifiedin block 401. In some embodiments, the predictive analysis server 111generates the first randomly-selected predictive model based on theweighted random selection for the predictive models identified in block402. For example, to generate the first randomly-selected predictivemodel, the predictive analysis server 111 may randomly select apredictive model among the predictive model identified in block 402 asthe first randomly-selected predictive model, where the random selectionmay be performed based on a probability distribution determined based onthe random selection parameters for the weighted random selection forthe predictive models.

At block 602, the predictive analysis server 111 determines a firstcandidate cross-model predictive score for the first randomly-selectedpredictive model. In some embodiments, to determine the first candidatecross-model predictive score, the predictive analysis server 111identifies the cross-model predictive score for the predictive inputgenerated by the first randomly-selected predictive model and determinesthe first candidate cross-model predictive score by applying the firstrandomly-selected predictive model to the predictive input using togenerate one or more first candidate model-specific predictive scoresand subsequently performs cross-model normalization on the one or morefirst candidate model-specific predictive scores to generate the firstcandidate cross-model predictive score.

At block 603, the predictive analysis server 111 determines whether thefirst candidate cross-model predictive score for the predictive inputsatisfies cross-model score adoption data for the predictive modelsidentified in block 402. In some embodiments, the cross-model scoreadoption data may indicate whether the predictive analysis server 111should adopt the first candidate cross-model predictive score for thepredictive input as the cross-model predictive score for the predictiveinput, e.g., based on one or more properties of the predictive inputand/or one or more properties of the first candidate cross-modelpredictive score.

At block 604, in response to determining that the first candidatecross-model predictive score for the predictive input satisfies thecross-model score adoption data, the predictive analysis server 111adopts the first candidate cross-model predictive score as thecross-model predictive score for the predictive input. At block 605, inresponse to determining that the first candidate cross-model predictivescore for the predictive input fails to satisfy the cross-model scoreadoption data, the predictive analysis server 111 selects an alternativecross-model predictive score as the cross-model predictive score. Insome embodiments, in response to determining that the first candidatecross-model predictive score for the predictive input fails to satisfythe cross-model score adoption data, the predictive analysis server 111modifies the predictive inputs to eliminate the predictive input.

In some embodiments, to determine an alternative cross-model predictivescore, the predictive analysis server 111 determines a second candidatecross-model predictive score by applying a champion predictive model tothe predictive input to generate one or more second candidatemodel-specific predictive scores and performing cross-model on the oneor more second candidate model-specific predictive scores to generatethe second candidate cross-model predictive score. The predictiveanalysis server 111 then determines whether the second candidatecross-model predictive score for the predictive input satisfiescross-model score adoption data for the predictive models. In responseto determining that the second candidate cross-model predictive scorefor the predictive input satisfies the cross-model score adoption data,the predictive analysis server 111 adopts the second candidatecross-model predictive score as the cross-model predictive score for thepredictive input.

In some of those embodiments, in response to determining that the secondcandidate cross-model predictive score for the predictive input fails tosatisfy the cross-model score adoption data, the predictive analysisserver 111 performs a second weighted random selection for thepredictive input to determine a third randomly-selected predictive modelof the predictive models. In an embodiment, the predictive analysisserver 111 determines a third candidate cross-model predictive score byapplying the third randomly-selected predictive model to the predictiveinput to generate one or more third candidate model-specific predictivescores and performing cross-model normalization on the one or more thirdcandidate model-specific predictive scores to generate the secondcandidate cross-model predictive score. In another embodiment, thepredictive analysis server 111 determines whether the third candidatecross-model predictive score for the predictive input satisfiescross-model score adoption data for the predictive models. Additionally,in certain embodiments in response to determining that the thirdcandidate cross-model predictive score for the predictive inputsatisfies the cross-model score adoption data for the predictive models,the predictive analysis server 111 adopts the third candidatecross-model predictive score as the cross-model predictive score for thepredictive input.

In some embodiments, to determine an alternative cross-model predictivescore, the predictive analysis server 111 performs a second weightedrandom selection for the predictive input to determine a secondrandomly-selected predictive model of the predictive models. Thepredictive analysis server 111 then determines a second candidatecross-model predictive score by applying a second randomly-selectedpredictive model to the predictive input to generate one or more secondcandidate model-specific predictive scores and performing cross-modelnormalization on the one or more second candidate model-specificpredictive scores to generate the second candidate cross-modelpredictive score. The predictive analysis server 111 then determineswhether the second candidate cross-model predictive score for thepredictive input satisfies cross-model score adoption data for thepredictive models. In response to determining that the second candidatecross-model predictive score for the predictive input satisfies thecross-model score adoption data for the predictive models, thepredictive analysis server 111 adopts the second candidate cross-modelpredictive score as the cross-model predictive score for the predictiveinput.

In some of those embodiments, in response to determining that the secondcandidate cross-model predictive score for the predictive input fails tosatisfy the cross-model score adoption data, the predictive analysisserver 111 identifies the cross-model predictive score for thepredictive input generated by the first randomly-selected predictivemodel. In an embodiment, the predictive analysis server 111 determines afirst candidate cross-model predictive score by applying the firstrandomly-selected predictive model to the predictive input to generateone or more first candidate model-specific predictive scores andperforming the cross-model normalization on the one or more firstcandidate model-specific predictive scores to generate the firstcandidate cross-model predictive score. In another embodiment, thepredictive analysis server 111 determines whether the first candidatecross-model predictive score for the predictive input satisfiescross-model score adoption data for the predictive models. Additionally,in certain embodiments in response to determining that the firstcandidate cross-model predictive score for the predictive inputsatisfies the cross-model score adoption data for the predictive models,the predictive analysis server 111 adopts the first candidatecross-model predictive score as the cross-model predictive score for thepredictive input.

Returning to FIG. 5 , the depicted process continues at block 502 whenthe predictive analysis server 111 determines one or more model-specificpredictive scores for the predictive input by applying the respectiveselected predictive model for the predictive input to the predictiveinput. In some embodiments, the predictive analysis server 111 providesthe predictive input as an input to the selected predictive models,where the selected predictive model is configured to process thepredictive input to generate the one or more model-specific predictivescores. In some embodiments, each of the one or more model-specificpredictive scores has a model-specific range that may be different froma cross-model range of a cross-model predictive score determined in thesubsequent block 503.

At block 503, the predictive analysis server 111 determines across-model predictive score for the predictive input by performingcross-model normalization on the one or more model-specific predictivescores associated with the predictive input. In some embodiments, thepredictive analysis server 111 performs cross-model normalization on theone or more mode-specific predictive scores to generate a cross-modelpredictive score. In some embodiments, the predictive analysis server111 performs cross-model normalization on the one or more model-specificpredictive scores generated by a selected predictive model based on thecross-model normalization data for the selected predictive model. Insome embodiments, each cross-model normalization data for a predictivemodel defines one or more cross-model conversion operations for thepredictive model. In some embodiments, the one or more cross-modelconversion operations for a respective predictive model are configuredto convert the cross-model predictive output for the respectivepredictive model each having an output-specific range to a cross-modelvalue having a cross-model range. In some embodiments, each cross-modelpredictive score generated by cross-model normalization is determinedbased on a respective cross-model value generated based on thecross-model normalization data associated with the cross-modelnormalization.

Returning to FIG. 4 , the process 400 continues at block 406 when thepredictive analysis server 111 determines, based on each cross-modelpredictive score for a predictive input obtained in block 401, across-model predictive output. In some embodiments, to determine across-model predictive output based on each cross-model predictivescore, the predictive analysis server 111 determines a predictiveranking of the predictive inputs based on each cross-model predictivescore associated with a predictive input. Additionally, in someembodiments, the predictive analysis server 111 determines thecross-model predictive output based on the predictive ranking of thepredictive inputs. For example, the predictive analysis server 111 mayrandomly select a predictive input of the top m predictive inputs in thepredictive ranking, where m may for example be equal to a ratio (e.g.,one-fourth) of a total number of predictive inputs and/or may be equalto a constant number (e.g., five). In certain embodiments, a predictiveranking can be a ranking of merchants that provide goods and/orservices. In certain embodiments, an electronic communication for amerchant promotional outreach can be transmitted to a consumer device(e.g., a client computing device from the client computing devices 101)to facilitate rendering of data associated with the merchant promotionaloutreach via an electronic interface (e.g., a graphical user interface)of the consumer device. For example, in certain embodiments, anelectronic communication related to a merchant promotional outreach fora merchant of goods or services can be generated based on thecross-model predictive output and/or the predictive ranking. In certainembodiments, the cross-model predictive output can indicate one ormerchants (e.g., one or more merchant identifiers) for a merchantpromotional outreach. Furthermore, the electronic communication relatedto the merchant promotional outreach can be transmitted to a consumerdevice (e.g., a client computing device from the client computingdevices 101) to facilitate rendering data associated with the merchantpromotional outreach via an electronic interface (e.g., a graphical userinterface) of the consumer device. In certain embodiments, thecross-model predictive output can be additionally or alternativelyemployed to update one or more predictive models. For example, one ormore predictive models can be retrained based on the cross-modelpredictive output. In certain embodiments, random selection ofpredictive input can facilitate a decrease in a number of computingresources for generating a predictive ranking.

In some embodiments, the predictive analysis server 111 identifies achallenger predictive model of the predictive models identified in block402. In some of those embodiments, the predictive analysis server 111identifies the champion predictive model includes identifying, for eachpredictive model of the identified predictive models, a retrospectivepredictive success score. The retrospective predictive success score foreach predictive model can, for example, define a retrospectiveprediction-outcome correspondence for the predictive model between oneor more retrospective predictive scores generated by the predictivemodel and one or more retrospective ground-truth predictive outcomescorresponding to the one or more past predictive scores. In anembodiment, the predictive analysis server 111 determines the championpredictive model based on each retrospective predictive success scoreassociated with a predictive model of the identified predictive models.

In some embodiments, each predictive input of the plurality ofpredictive inputs is associated with a candidate merchant identifier. Inan aspect, each candidate merchant identifier of the plurality ofcandidate merchant identifiers can be associated with a candidatemerchant data structure. In another aspect, each predictive inputassociated with a candidate merchant identifier comprises the candidatemerchant data structure associated with the candidate merchantidentifier. Additionally, each cross-model predictive score for apredictive input of the plurality of predictive inputs can indicate amerchant promotional outreach interest prediction for the candidatemerchant identifier associated with the predictive input. Thecross-model predictive output can indicate, for example, a thresholdnumber of merchant identifiers whose corresponding merchant promotionaloutreach interest prediction is highest among each merchant promotionaloutreach interest prediction associated with a candidate merchantidentifier.

Cross-Model Score Verification

FIG. 7 is an example flow diagram illustrating performing cross-modelscore verification for a predictive input. The process 700 illustratedin FIG. 7 is described herein with reference to a predictive analysisserver, such as the predictive analysis server 111 of FIG. 1 , but canbe performed by any system of one or more computers, such as a systemthat includes the predictive analysis server 111 of FIG. 1 . Each blockof the flowchart, and combinations of blocks in the flowchart, may beimplemented by various means such as hardware, firmware, circuitryand/or other devices associated with execution of software including oneor more computer program instructions. For example, one or more of theprocedures described in FIG. 7 may be embodied by computer programinstructions, which may be stored by a non-transitory memory of anapparatus employing an embodiment of the present disclosure and executedby a processor in the apparatus. These computer program instructions maydirect a computer or other programmable apparatus to function in aparticular manner, such that the instructions stored in thecomputer-readable storage memory produce an article of manufacture, theexecution of which implements the function specified in the flowchartblock(s).

As depicted in FIG. 7 , the process 700 starts at block 701 when thepredictive analysis server 111 identifies one or more unselectedpredictive models for the predictive input. In some embodiments, todetermine the one or more unselected predictive models for thepredictive input, the predictive analysis server 111 first identifies aselected predictive model for the predictive input (e.g., by determiningthe selected predictive model for the predictive input as described withreference to block 501 of FIG. 5 and/or with reference to various blocksof FIG. 6 ). Subsequent to identifying the selected predictive model,the predictive analysis server 111 determines any identified predictivemodels that are not designated as the selected predictive model for thepredictive input as unselected predictive models for the predictiveinput.

At block 702, the predictive analysis server 111 determines, for eachunselected predictive model of the one or more unselected predictivemodels, a verification predictive score by applying the unselectedpredictive model to the predictive input. In some embodiments, thepredictive analysis server 111 applies each unselected predictive modelfor a predictive input to the predictive input to determine acorresponding verification predictive score. In some embodiments, thepredictive analysis server 111 applies each unselected predictive modelfor a predictive input to a predictive input to determine acorresponding model-based verification predictive score, appliescross-model normalization for an unselected predictive model to thecorresponding model-based verification predictive score for theunselected predictive model to determine a corresponding cross-modelverification predictive score, and determines each correspondingverification predictive score for an unselected predictive model basedon the corresponding cross-model verification predictive score for theunselected predictive model (e.g., adopts each corresponding cross-modelverification predictive score for an unselected predictive model as thecorresponding verification predictive score for the unselectedpredictive model).

At block 703, the predictive analysis server 111 obtains a prospectiveground-truth success outcome for the predictive input corresponding tothe threshold prospective performance time interval. The predictiveanalysis server 111 can obtain the prospective ground-truth successoutcome, for example, subsequent to expiration of a thresholdprospective performance time interval after determining the cross-modelpredictive output for a predictive input. In certain embodiments, theprospective ground-truth success outcome can describe an observedmerchant promotional outreach interest prediction. The observed merchantpromotional outreach interest prediction can, for example, be based onmerchant outreach during the threshold prospective performance timeinterval.

At block 704, the predictive analysis server 111 determines aprospective predictive success score for each predictive model of theplurality of predictive models. The prospective predictive success scorefor the selected predictive model can be determined, for example, basedon a recent ground-truth success outcome for the predictive input andthe cross-model predictive score for the predictive input. Furthermore,each prospective predictive success score for an unselected predictivemodel of the one or more unselected predictive model can be determinedbased on the recent ground-truth success outcome for the predictiveinput and the verification predictive score associated with theunselected predictive model.

In some embodiments, the predictive analysis server adjusts theselection probability score for each predictive model based on theprospective predictive success score for the predictive model. In someembodiments, each cross-model score generation is performed by a firstcomputing device configured to perform the cross-model score generationbased on one or more first performance metrics. In some embodiments,each cross-model score verification is performed by a second computingdevice configured to perform each cross-model score verification basedon one or more second performance metrics. In some embodiments, at leastone first performance metric of the one or more first performancemetrics requires greater end-user response speed than at least onesecond performance metric of the one or more second performance metrics.In some embodiments, the one or more front-end performance metricscomprise one or more real-time performance metrics.

Exemplary Electronic Interfaces

FIG. 8 is an example electronic interface 800 illustrating performancemetrics of merchant acquisition rankings. The electronic interface 800can be, for example, an electronic interface (e.g., a graphical userinterface) of a consumer device. In an embodiment, the electronicinterface 800 can be an electronic dashboard that renders one or moregraphical elements associated with performance metrics. As illustratedin FIG. 8 , the electronic interface 800 includes performance metricsdata 802 for a first predictive model, performance metrics data 804 fora second predictive model, performance metrics data 806 for a thirdpredictive model, and performance metrics data 808 for a fourthpredictive model. In certain embodiments, the performance metrics data802, the performance metrics data 804, the performance metrics data 806,and/or the performance metrics data 808 can be presented based on one ormore intervals of time. As such, the electronic interface 800 can beemployed to more efficiently and/or more accurately identify topmerchants and/or optimal predictive models for different metrics.

FIG. 9 is an example electronic interface illustrating performancemetrics of merchants based on location. The electronic interface 900 canbe, for example, an electronic interface (e.g., a graphical userinterface) of a consumer device. In an embodiment, the electronicinterface 900 can be an electronic dashboard that renders one or moregraphical elements associated with performance metrics. As illustratedin FIG. 9 , the electronic interface 900 includes performance metricdata 902 associated with different locations, performance metric data904 associated with trends during different time periods, andperformance metric data 904 associated with revenue. In certainembodiments, the electronic interface 900 can include a set ofinteractive elements 908 that can facilitate modification of a type ofperformance metric rendered via the performance metric data 902, theperformance metric data 904 and/or the performance metric data 906. Assuch, the electronic interface 900 can be employed to more efficientlyand/or more accurately identify top merchants, objectives, categoriesrelated to goods or services, locations, and/or other metrics forpredictive models.

FIG. 10 is an example electronic interface 1000 illustrating performancemetrics of merchant acquisition rankings. The electronic interface 1000can be, for example, an electronic interface (e.g., a graphical userinterface) of a consumer device. In an embodiment, the electronicinterface 1000 can be an electronic dashboard that renders one or moregraphical elements associated with performance metrics. The electronicinterface 1000 can present performance metrics related to predictivemodels. As illustrated in FIG. 10 , the electronic interface 1000includes performance metrics data 1002 associated with predictive modelsbased on performance metric scores (e.g., a Spearman correlation score)for respective predictive models related to a ranking of merchants for aparticular market of goods or services. The electronic interface 1000also includes performance metrics data 1004 associated with predictivemodels based on performance metric scores (e.g., area under curve ratioscores) for respective predictive models related to a ranking ofmerchants for a particular market of goods or services. Furthermore, theelectronic interface 1000 includes performance metrics data 1006associated with predictive models based on performance metric scores(e.g., a Spearman correlation score) for respective predictive modelsrelated to a ranking of new merchants. The electronic interface 1000also includes performance metrics data 1008 associated with predictivemodels based on performance metric scores (e.g., area under curve ratioscores) for respective predictive models related to a ranking of newmerchants. In certain embodiments, the performance metrics data 1002,the performance metrics data 1004, the performance metrics data 1006,and/or the performance metrics data 1008 can be presented based on oneor more intervals of time. As such, the electronic interface 1000 can beemployed to more efficiently and/or more accurately identify topmerchants and/or optimal predictive models for different metrics.

FIG. 11 is an example electronic interface 1100 illustrating performancemetrics of merchant acquisition rankings. The electronic interface 1100can be, for example, an electronic interface (e.g., a graphical userinterface) of a consumer device. In an embodiment, the electronicinterface 1100 can be an electronic dashboard that renders one or moregraphical elements associated with performance metrics. The electronicinterface 1100 can present performance metrics related to predictivemodels. As illustrated in FIG. 11 , the electronic interface 1100includes performance metrics data 1102 associated with predictive modelsbased on performance metric scores (e.g., a Spearman correlation score)for respective predictive models related to a ranking of merchants for aparticular market of goods or services. The electronic interface 1100also includes performance metrics data 1104 associated with predictivemodels based on performance metric scores (e.g., area under curve ratioscores) for respective predictive models related to a ranking ofmerchants for a particular market of goods or services. Furthermore, theelectronic interface 1100 includes performance metrics data 1106associated with predictive models based on performance metric scores(e.g., a Spearman correlation score) for respective predictive modelsrelated to a ranking of new merchants. The electronic interface 1100also includes performance metrics data 1108 associated with predictivemodels based on performance metric scores (e.g., area under curve ratioscores) for respective predictive models related to a ranking of newmerchants. In certain embodiments, the performance metrics data 1102,the performance metrics data 1104, the performance metrics data 1106,and/or the performance metrics data 1108 can be presented based on oneor more intervals of time. As such, the electronic interface 1100 can beemployed to more efficiently and/or more accurately identify topmerchants and/or optimal predictive models for different metrics.

FIG. 12 is an example electronic interface 1200 illustrating performancemetrics of merchant acquisition rankings. The electronic interface 1200can be, for example, an electronic interface (e.g., a graphical userinterface) of a consumer device. In an embodiment, the electronicinterface 1200 can be an electronic dashboard that renders one or moregraphical elements associated with performance metrics. The electronicinterface 1200 can present performance metrics related to predictivemodels. As illustrated in FIG. 12 , the electronic interface 1200includes performance metrics data 1202 associated with predictive modelsbased on performance metric scores (e.g., a percentage ranking vs.cumulative new merchants) for respective predictive models related to aranking of new merchants. The electronic interface 1200 also includesperformance metrics data 1204 associated with predictive models based onperformance metric scores (e.g., a percentage ranking vs. cumulative newmerchants) for respective predictive models related to a ranking of newmerchants. In an embodiment, the electronic interface 1200 can include apredictive model ranking 1206 related to the performance data 1202. Forexample, the predictive model ranking 1206 can rank predictive modelsbased on respective score values and/or a respective type of performancemetric. Furthermore, the electronic interface 1200 can include apredictive model ranking 1208 related to the performance data 1204. Forexample, the predictive model ranking 1208 can rank predictive modelsbased on respective score values and/or a respective type of performancemetric. As such, the electronic interface 1200 can be employed to moreefficiently and/or more accurately identify top merchants and/or optimalpredictive models for different metrics.

Additional Implementation Details

Although example processing systems have been described in FIGS. 1-3 ,implementations of the subject matter and the functional operationsdescribed herein can be implemented in other types of digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them.

Embodiments of the subject matter and the operations described hereincan be implemented in digital electronic circuitry, or in computersoftware, firmware, or hardware, including the structures disclosed inthis specification and their structural equivalents, or in combinationsof one or more of them. Embodiments of the subject matter describedherein can be implemented as one or more computer programs, i.e., one ormore modules of computer program instructions, encoded oncomputer-readable storage medium for execution by, or to control theoperation of, information/data processing apparatus. Alternatively, orin addition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, which is generated toencode information/data for transmission to suitable receiver apparatusfor execution by an information/data processing apparatus. Acomputer-readable storage medium can be, or be included in, acomputer-readable storage device, a computer-readable storage substrate,a random or serial access memory array or device, or a combination ofone or more of them. Moreover, while a computer-readable storage mediumis not a propagated signal, a computer-readable storage medium can be asource or destination of computer program instructions encoded in anartificially-generated propagated signal. The computer-readable storagemedium can also be, or be included in, one or more separate physicalcomponents or media (e.g., multiple CDs, disks, or other storagedevices).

The operations described herein can be implemented as operationsperformed by an information/data processing apparatus oninformation/data stored on one or more computer-readable storage devicesor received from other sources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing. The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (Application Specific Integrated Circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor information/data (e.g., one or more scripts stored in a markuplanguage document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub-programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described herein can be performed by oneor more programmable processors executing one or more computer programsto perform actions by operating on input information/data and generatingoutput. Processors suitable for the execution of a computer programinclude, by way of example, both general and special purposemicroprocessors, and any one or more processors of any kind of digitalcomputer. Generally, a processor will receive instructions andinformation/data from a read-only memory, a random access memory, orboth. The essential elements of a computer are a processor forperforming actions in accordance with instructions and one or morememory devices for storing instructions and data. Generally, a computerwill also include, or be operatively coupled to receive information/datafrom or transfer information/data to, or both, one or more mass storagedevices for storing data, e.g., magnetic, magneto-optical disks, oroptical disks. However, a computer need not have such devices. Devicessuitable for storing computer program instructions and information/datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described herein can be implemented on a computer having adisplay device, e.g., a CRT (cathode ray tube) or LCD (liquid crystaldisplay) monitor, for displaying information/data to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser’s client computing device in response to requests received from theweb browser.

Embodiments of the subject matter described herein can be implemented ina computing system that includes a back-end component, e.g., as aninformation/data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computing device having a graphical user interface or a webbrowser through which a user can interact with an implementation of thesubject matter described herein, or any combination of one or more suchback-end, middleware, or front-end components. The components of thesystem can be interconnected by any form or medium of digitalinformation/data communication, e.g., a communication network. Examplesof communication networks include a local area network (“LAN”) and awide area network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits information/data (e.g., an HTML page) toa client computing device (e.g., for purposes of displayinginformation/data to and receiving user input from a user interactingwith the client computing device). Information/data generated at theclient computing device (e.g., a result of the user interaction) can bereceived from the client computing device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anydisclosures or of what may be claimed, but rather as description offeatures specific to particular embodiments of particular disclosures.Certain features that are described herein in the context of separateembodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable sub-combination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asub-combination or variation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults, unless described otherwise. In certain circumstances,multitasking and parallel processing may be advantageous. Moreover, theseparation of various system components in the embodiments describedabove should not be understood as requiring such separation in allembodiments, and it should be understood that the described programcomponents and systems can generally be integrated together in a singlesoftware product or packaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults, unless described otherwise. In certain implementations,multitasking and parallel processing may be advantageous.

Conclusion

Many modifications and other embodiments of the disclosures set forthherein will come to mind to one skilled in the art to which thesedisclosures pertain having the benefit of the teachings presented in theforegoing description and the associated drawings. Therefore, it is tobe understood that the disclosures are not to be limited to the specificembodiments disclosed and that modifications and other embodiments areintended to be included within the scope of the appended claims.Although specific terms are employed herein, they are used in a genericand descriptive sense only and not for purposes of limitation, unlessdescribed otherwise.

1-22. (canceled)
 23. An apparatus comprising at least one processor andat least one non-transitory memory storing instructions that, with theat least one processor, cause the apparatus to: determine a cross-modelpredictive score for each predictive input of a plurality of predictiveinputs by: determining, based at least in part on a weighted randomselection for the predictive input, a respective selected predictivemodel of a plurality of predictive models, wherein at least one weightedrandom selection parameter is determined based at least in part onrespective selection probability scores of the plurality of predictivemodels; generating, based at least in part on one or more predictiveoperations of the respective selected predictive model and thepredictive input, one or more model-specific predictive scores for thepredictive input; and determining the cross-model predictive score forthe predictive input based at least in part on transforming the one ormore model-specific predictive scores associated with the predictiveinput using respective cross-model normalization parameters ofcross-model normalization data for the selected predictive modelassociated with the predictive input; generate, based at least in parton one or more optimal predictive models and one or more predictiveinputs of the predictive ranking of predictive inputs, cross-modelpredictive output; generate, based at least in part on the cross-modelpredictive output, an electronic communication; and transmit theelectronic communication to a computing device, wherein the electroniccommunication is configured to render data via a graphical interface ofthe computing device.
 24. The apparatus of claim 23, wherein the atleast one non-transitory memory stores instructions that, with the atleast one processor, further cause the apparatus to: obtain a modelselection probability distribution, wherein the model selectionprobability distribution defines, for each predictive model of theplurality of predictive models, the respective selection probabilityscore.
 25. The apparatus of claim 23, wherein the at least onenon-transitory memory stores instructions that, with the at least oneprocessor, further cause the apparatus to: obtain, for each predictivemodel of the plurality of predictive models, the respective cross-modelnormalization data.
 26. The apparatus of claim 23, wherein the at leastone non-transitory memory stores instructions that, with the at leastone processor, further cause the apparatus to: determine, based at leastin part on a predictive ranking of predictive inputs of the plurality ofpredictive inputs ranked according to respective cross-model predictivescores for the predictive inputs, the one or more optimal predictivemodels of the plurality of predictive models.
 27. The apparatus of claim23, wherein the electronic communication is associated with a merchantof goods or services.
 28. The apparatus of claim 23, wherein theweighted random selection for a predictive input of the plurality ofpredictive inputs is characterized by one or more weighted randomselection parameters for the predictive input.
 29. The apparatus ofclaim 23, wherein the at least one non-transitory memory storesinstructions that, with the at least one processor, further cause theapparatus to: identify, from the plurality of predictive models, achampion predictive model and one or more challenger predictive models.30. A computer-implemented method, comprising: determining a cross-modelpredictive score for each predictive input of a plurality of predictiveinputs by: determining, based at least in part on a weighted randomselection for the predictive input, a respective selected predictivemodel of a plurality of predictive models, wherein at least one weightedrandom selection parameter is determined based at least in part onrespective selection probability scores of the plurality of predictivemodels; generating, based at least in part on one or more predictiveoperations of the respective selected predictive model and thepredictive input, one or more model-specific predictive scores for thepredictive input; and determining the cross-model predictive score forthe predictive input based at least in part on transforming the one ormore model-specific predictive scores associated with the predictiveinput using respective cross-model normalization parameters ofcross-model normalization data for the selected predictive modelassociated with the predictive input; generating, based at least in parton one or more optimal predictive models and one or more predictiveinputs of the predictive ranking of predictive inputs, cross-modelpredictive output; generating, based at least in part on the cross-modelpredictive output, an electronic communication; and transmitting theelectronic communication to a computing device, wherein the electroniccommunication is configured to render data via a graphical interface ofthe computing device.
 31. The method of claim 30, further comprising:obtaining a model selection probability distribution, wherein the modelselection probability distribution defines, for each predictive model ofthe plurality of predictive models, the respective selection probabilityscore.
 32. The method of claim 30, further comprising: obtaining, foreach predictive model of the plurality of predictive models, therespective cross-model normalization data.
 33. The method of claim 30,further comprising: determining, based at least in part on a predictiveranking of predictive inputs of the plurality of predictive inputsranked according to respective cross-model predictive scores for thepredictive inputs, the one or more optimal predictive models of theplurality of predictive models.
 34. The method of claim 30, wherein theelectronic communication is associated with a merchant of goods orservices.
 35. The method of claim 30, wherein the weighted randomselection for a predictive input of the plurality of predictive inputsis characterized by one or more weighted random selection parameters forthe predictive input.
 36. The method of claim 30, further comprising:identifying, from the plurality of predictive models, a championpredictive model and one or more challenger predictive models.
 37. Atleast one non-transitory memory storing instructions that, with at leastone processor, cause an apparatus to: determine a cross-model predictivescore for each predictive input of a plurality of predictive inputs by:determining, based at least in part on a weighted random selection forthe predictive input, a respective selected predictive model of aplurality of predictive models, wherein at least one weighted randomselection parameter is determined based at least in part on respectiveselection probability scores of the plurality of predictive models;generating, based at least in part on one or more predictive operationsof the respective selected predictive model and the predictive input,one or more model-specific predictive scores for the predictive input;and determining the cross-model predictive score for the predictiveinput based at least in part on transforming the one or moremodel-specific predictive scores associated with the predictive inputusing respective cross-model normalization parameters of cross-modelnormalization data for the selected predictive model associated with thepredictive input; generate, based at least in part on one or moreoptimal predictive models and one or more predictive inputs of thepredictive ranking of predictive inputs, cross-model predictive output;generate, based at least in part on the cross-model predictive output,an electronic communication; and transmit the electronic communicationto a computing device, wherein the electronic communication isconfigured to render data via a graphical interface of the computingdevice.
 38. The at least one non-transitory memory of claim 37, whereinthe at least one non-transitory memory stores instructions that, withthe at least one processor, further cause the apparatus to: obtain, foreach predictive model of the plurality of predictive models, therespective cross-model normalization data.
 39. The at least onenon-transitory memory of claim 37, wherein the at least onenon-transitory memory stores instructions that, with the at least oneprocessor, further cause the apparatus to: determine, based at least inpart on a predictive ranking of predictive inputs of the plurality ofpredictive inputs ranked according to respective cross-model predictivescores for the predictive inputs, the one or more optimal predictivemodels of the plurality of predictive models.
 40. The at least onenon-transitory memory of claim 37, wherein the electronic communicationis associated with a merchant of goods or services.
 41. The at least onenon-transitory memory of claim 37, wherein the weighted random selectionfor a predictive input of the plurality of predictive inputs ischaracterized by one or more weighted random selection parameters forthe predictive input.
 42. The at least one non-transitory memory ofclaim 37, wherein the at least one non-transitory memory storesinstructions that, with the at least one processor, further cause theapparatus to: identify, from the plurality of predictive models, achampion predictive model and one or more challenger predictive models.